[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:04:56 dist-test-slave-7g8g krb5kdc[12304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:04:56 dist-test-slave-7g8g krb5kdc[12304](info): set up 2 sockets
Jun 20 23:04:56 dist-test-slave-7g8g krb5kdc[12304](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 23:04:57.642256 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.006s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:04:57 dist-test-slave-7g8g krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:04:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:04:57.812173 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:34679
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:34679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:58.113847 12317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:58.114125 12317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:58.114215 12317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:58.121191 12317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:58.121282 12317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:58.121330 12317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:58.121385 12317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:58.121428 12317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:58.130726 12317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:34679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:34679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:04:58.131953 12317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:58.133200 12317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:58.143975 12322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:58.144344 12323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:58.144588 12325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:58.144984 12317 server_base.cc:1034] running on GCE node
I20260620 23:04:58.145762 12317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:58.147130 12317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:58.148325 12317 hybrid_clock.cc:648] HybridClock initialized: now 1781996698148273 us; error 40 us; skew 500 ppm
I20260620 23:04:58.148654 12317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:04:58 dist-test-slave-7g8g krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996698, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:58.151481 12317 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:04:58.152889 12317 webserver.cc:460] Webserver started at http://127.12.2.126:45745/ using document root <none> and password file <none>
I20260620 23:04:58.153528 12317 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:58.153643 12317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:58.153918 12317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:58.156764 12317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "e8d961c302ba495d94115ee74d5790aa"
format_stamp: "Formatted at 2026-06-20 23:04:58 on dist-test-slave-7g8g"
server_key: "c4c03b4e7b7941ed22b224b0c1d6a643"
server_key_iv: "55909b9fe724e1fe9c89ca852fd72742"
server_key_version: "encryptionkey@0"
I20260620 23:04:58.157439 12317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "e8d961c302ba495d94115ee74d5790aa"
format_stamp: "Formatted at 2026-06-20 23:04:58 on dist-test-slave-7g8g"
server_key: "c4c03b4e7b7941ed22b224b0c1d6a643"
server_key_iv: "55909b9fe724e1fe9c89ca852fd72742"
server_key_version: "encryptionkey@0"
I20260620 23:04:58.162213 12317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 23:04:58.165396 12332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:58.166658 12317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:04:58.166805 12317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "e8d961c302ba495d94115ee74d5790aa"
format_stamp: "Formatted at 2026-06-20 23:04:58 on dist-test-slave-7g8g"
server_key: "c4c03b4e7b7941ed22b224b0c1d6a643"
server_key_iv: "55909b9fe724e1fe9c89ca852fd72742"
server_key_version: "encryptionkey@0"
I20260620 23:04:58.167007 12317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:58.180130 12317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:58.183826 12317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:58.184057 12317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:58.200130 12317 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:34679
I20260620 23:04:58.200155 12383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:34679 every 8 connection(s)
I20260620 23:04:58.201421 12317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:04:58.205183 12384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:58.205583 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 12317
I20260620 23:04:58.205850 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:04:58.207674 12297 external_mini_cluster.cc:1428] Setting key eeea116451536bc708980e9aebfc8c69
I20260620 23:04:58.216506 12384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa: Bootstrap starting.
I20260620 23:04:58.220391 12384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:58.221609 12384 log.cc:824] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:58.224797 12384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa: No bootstrap required, opened a new log
I20260620 23:04:58.233834 12384 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d961c302ba495d94115ee74d5790aa" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34679 } }
I20260620 23:04:58.234206 12384 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:58.234313 12384 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d961c302ba495d94115ee74d5790aa, State: Initialized, Role: FOLLOWER
I20260620 23:04:58.234985 12384 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d961c302ba495d94115ee74d5790aa" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34679 } }
I20260620 23:04:58.235287 12384 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:04:58.235421 12384 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:04:58.235587 12384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:58.238534 12384 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d961c302ba495d94115ee74d5790aa" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34679 } }
I20260620 23:04:58.239112 12384 leader_election.cc:302] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8d961c302ba495d94115ee74d5790aa; no voters:
I20260620 23:04:58.239763 12384 leader_election.cc:288] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:04:58.239904 12389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [term 1 FOLLOWER]: Leader election won for term 1
Jun 20 23:04:58 dist-test-slave-7g8g krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:04:58.241169 12389 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [term 1 LEADER]: Becoming Leader. State: Replica: e8d961c302ba495d94115ee74d5790aa, State: Running, Role: LEADER
I20260620 23:04:58.241672 12389 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d961c302ba495d94115ee74d5790aa" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34679 } }
I20260620 23:04:58.242103 12384 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:58.245471 12391 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8d961c302ba495d94115ee74d5790aa. Latest consensus state: current_term: 1 leader_uuid: "e8d961c302ba495d94115ee74d5790aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d961c302ba495d94115ee74d5790aa" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34679 } } }
I20260620 23:04:58.246003 12391 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:58.245573 12390 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8d961c302ba495d94115ee74d5790aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d961c302ba495d94115ee74d5790aa" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34679 } } }
I20260620 23:04:58.246871 12395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:04:58.246959 12390 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:58.256359 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:58.211378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.12.2.126:34679)
0620 23:04:58.212188 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:58.212203 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:58.214906 (+ 2703us) server_negotiation.cc:407] Connection header received
0620 23:04:58.225941 (+ 11035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:58.225962 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:58.226336 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:58.226666 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:58.230292 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:58.231654 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:58.233951 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:58.234685 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:58.234815 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:58.241759 (+ 6944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:58.241770 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:58.241824 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:58.241880 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:58.246663 (+ 4783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:58.247661 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:58.247670 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:58.247677 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:58.247875 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:58.248463 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:58.248468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:58.248471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:58.249211 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:58.252275 (+ 3064us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:58.252523 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 23:04:58.252954 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":343,"threads_started":1}
I20260620 23:04:58.258013 12395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:04:58.267174 12395 catalog_manager.cc:1347] Generated new cluster ID: 636c82e5122346768c84aae00e509dc6
I20260620 23:04:58.267359 12395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:04:58.288123 12395 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:04:58.289664 12395 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:04:58.307472 12395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8d961c302ba495d94115ee74d5790aa: Generated new TSK 0
I20260620 23:04:58.308465 12395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:04:58.356194 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34679
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:58.661254 12410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:58.661509 12410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:58.661602 12410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:58.668767 12410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:58.668875 12410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:58.668998 12410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:04:58.678480 12410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34679
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:04:58.679773 12410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:58.681037 12410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:58.693010 12415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:58.693138 12416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:58.693279 12418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:58.694391 12410 server_base.cc:1034] running on GCE node
I20260620 23:04:58.694916 12410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:58.695815 12410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:58.697017 12410 hybrid_clock.cc:648] HybridClock initialized: now 1781996698696966 us; error 34 us; skew 500 ppm
I20260620 23:04:58.697307 12410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:04:58 dist-test-slave-7g8g krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996698, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:58.700253 12410 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:04:58.701777 12410 webserver.cc:460] Webserver started at http://127.12.2.65:34883/ using document root <none> and password file <none>
I20260620 23:04:58.702435 12410 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:58.702564 12410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:58.702872 12410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:58.705777 12410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "24627fb9340a4267880fa0a23021c331"
format_stamp: "Formatted at 2026-06-20 23:04:58 on dist-test-slave-7g8g"
server_key: "5c08447c80c70df3e973bc7ab7545cf1"
server_key_iv: "12e070381a8441e21d5c2d271c0d408e"
server_key_version: "encryptionkey@0"
I20260620 23:04:58.706601 12410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "24627fb9340a4267880fa0a23021c331"
format_stamp: "Formatted at 2026-06-20 23:04:58 on dist-test-slave-7g8g"
server_key: "5c08447c80c70df3e973bc7ab7545cf1"
server_key_iv: "12e070381a8441e21d5c2d271c0d408e"
server_key_version: "encryptionkey@0"
I20260620 23:04:58.711689 12410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:04:58.714833 12425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:58.716092 12410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 23:04:58.716238 12410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "24627fb9340a4267880fa0a23021c331"
format_stamp: "Formatted at 2026-06-20 23:04:58 on dist-test-slave-7g8g"
server_key: "5c08447c80c70df3e973bc7ab7545cf1"
server_key_iv: "12e070381a8441e21d5c2d271c0d408e"
server_key_version: "encryptionkey@0"
I20260620 23:04:58.716400 12410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:58.729162 12410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:58.732630 12410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:58.732867 12410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:58.733688 12410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:58.734861 12410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:58.734953 12410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:58.735076 12410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:58.735132 12410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:58.763965 12410 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:36337
I20260620 23:04:58.764039 12537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:36337 every 8 connection(s)
I20260620 23:04:58.765246 12410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:04:58.770710 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 12410
I20260620 23:04:58.770962 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:04:58.771647 12297 external_mini_cluster.cc:1428] Setting key 76226e56aaed27d9c35996509d7e76db
Jun 20 23:04:58 dist-test-slave-7g8g krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996698, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:04:58.785758 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:58.767656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:33099 (local address 127.12.2.126:34679)
0620 23:04:58.768052 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:58.768056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:58.769003 (+ 947us) server_negotiation.cc:407] Connection header received
0620 23:04:58.771073 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:58.771077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:58.771237 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:58.771370 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:58.773139 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:58.773904 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:58.776051 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:58.776479 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:58.776570 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:58.779970 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:58.779976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:58.779981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:58.780015 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:58.782805 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:58.783492 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:58.783497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:58.783500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:58.783638 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:58.784071 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:58.784076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:58.784079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:58.784436 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:58.784581 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:58.785174 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 23:04:58.785392 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":195,"threads_started":1}
I20260620 23:04:58.786706 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:58.768197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34679 (local address 127.12.2.65:33099)
0620 23:04:58.768857 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:58.768898 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:58.770626 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:58.771637 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:58.771649 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:58.772338 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:58.772991 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:58.773014 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:58.774053 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:58.774058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:58.775916 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:58.775932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:58.776673 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:58.776679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:58.776921 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:58.777569 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:58.777596 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:58.779768 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:58.782970 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:58.782979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:58.782991 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:58.783383 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:58.783735 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:58.783738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:58.783740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:58.783956 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:58.784537 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:58.784542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:58.784833 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 23:04:58.785035 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 23:04:58.785524 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":265,"threads_started":1}
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
I20260620 23:04:58.788269 12538 heartbeater.cc:347] Connected to a master server at 127.12.2.126:34679
I20260620 23:04:58.788537 12538 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:58.789268 12538 heartbeater.cc:511] Master 127.12.2.126:34679 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
I20260620 23:04:58.791390 12349 ts_manager.cc:194] Registered new tserver with Master: 24627fb9340a4267880fa0a23021c331 (127.12.2.65:36337)
I20260620 23:04:58.792821 12349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:33099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:04:58.808138 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34679
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:59.109215 12544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:59.109483 12544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:59.109614 12544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:59.116606 12544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:59.116724 12544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:59.116878 12544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:04:59.126448 12544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34679
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:04:59.127732 12544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:59.128994 12544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:59.141229 12550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:59.141428 12552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:59.141811 12549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:59.142629 12544 server_base.cc:1034] running on GCE node
I20260620 23:04:59.143256 12544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:59.144131 12544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:59.145324 12544 hybrid_clock.cc:648] HybridClock initialized: now 1781996699145273 us; error 41 us; skew 500 ppm
I20260620 23:04:59.145642 12544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:04:59 dist-test-slave-7g8g krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996699, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:59.148494 12544 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:04:59.149869 12544 webserver.cc:460] Webserver started at http://127.12.2.66:38851/ using document root <none> and password file <none>
I20260620 23:04:59.150489 12544 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:59.150604 12544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:59.150884 12544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:59.153748 12544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "70d951187f834a2db49e8c634f5f0a69"
format_stamp: "Formatted at 2026-06-20 23:04:59 on dist-test-slave-7g8g"
server_key: "7678b93f6ca8c904a6384d909017de98"
server_key_iv: "8ac58b8e3c808ce99f4ce9e7baab5a5f"
server_key_version: "encryptionkey@0"
I20260620 23:04:59.154470 12544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "70d951187f834a2db49e8c634f5f0a69"
format_stamp: "Formatted at 2026-06-20 23:04:59 on dist-test-slave-7g8g"
server_key: "7678b93f6ca8c904a6384d909017de98"
server_key_iv: "8ac58b8e3c808ce99f4ce9e7baab5a5f"
server_key_version: "encryptionkey@0"
I20260620 23:04:59.159297 12544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260620 23:04:59.162572 12559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:59.163998 12544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:04:59.164119 12544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "70d951187f834a2db49e8c634f5f0a69"
format_stamp: "Formatted at 2026-06-20 23:04:59 on dist-test-slave-7g8g"
server_key: "7678b93f6ca8c904a6384d909017de98"
server_key_iv: "8ac58b8e3c808ce99f4ce9e7baab5a5f"
server_key_version: "encryptionkey@0"
I20260620 23:04:59.164285 12544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:59.190593 12544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:59.194839 12544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:59.195144 12544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:59.196102 12544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:59.197438 12544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:59.197528 12544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:59.197688 12544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:59.197763 12544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:59.227276 12544 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:36023
I20260620 23:04:59.227314 12671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:36023 every 8 connection(s)
I20260620 23:04:59.228425 12544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:04:59.231830 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 12544
I20260620 23:04:59.232010 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:04:59.232522 12297 external_mini_cluster.cc:1428] Setting key 5c5293154682e32e8c1267baba3df4b2
Jun 20 23:04:59 dist-test-slave-7g8g krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996699, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
I20260620 23:04:59.248601 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:59.230740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:41339 (local address 127.12.2.126:34679)
0620 23:04:59.230908 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:59.230913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:59.232175 (+ 1262us) server_negotiation.cc:407] Connection header received
0620 23:04:59.234301 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:59.234306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:59.234379 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:59.234462 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:59.235914 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.236897 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.238353 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.238981 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.239109 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:59.242443 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:59.242449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:59.242453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:59.242490 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:59.245192 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.246064 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.246071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.246074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.246223 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.246655 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.246660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.246662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.247007 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:59.247152 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:59.247897 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 23:04:59.248134 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
I20260620 23:04:59.249516 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:59.231241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34679 (local address 127.12.2.66:41339)
0620 23:04:59.232020 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:59.232073 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:59.234043 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:59.234675 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:59.234683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:59.235221 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:59.235761 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:59.235778 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.237035 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.237039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:59.238216 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:59.238228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.239217 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.239222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:59.239418 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:59.240053 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:59.240079 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:59.242245 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:59.245313 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:59.245322 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:59.245336 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:59.245921 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:59.246333 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:59.246339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:59.246343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:59.246542 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:59.247134 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:59.247142 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:59.247505 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 23:04:59.247776 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 23:04:59.248269 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":280,"threads_started":1}
I20260620 23:04:59.250926 12672 heartbeater.cc:347] Connected to a master server at 127.12.2.126:34679
I20260620 23:04:59.251248 12672 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:59.252007 12672 heartbeater.cc:511] Master 127.12.2.126:34679 requested a full tablet report, sending...
I20260620 23:04:59.253510 12349 ts_manager.cc:194] Registered new tserver with Master: 70d951187f834a2db49e8c634f5f0a69 (127.12.2.66:36023)
I20260620 23:04:59.254263 12349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:41339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:04:59.269496 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34679
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:59.577073 12677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:59.577364 12677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:59.577472 12677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:59.584735 12677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:59.584858 12677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:59.584988 12677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:04:59.594681 12677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34679
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:04:59.596046 12677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:59.597376 12677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:59.610289 12682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:59.610335 12685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:59.610253 12683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:59.611153 12677 server_base.cc:1034] running on GCE node
I20260620 23:04:59.611699 12677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:59.612560 12677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:59.613740 12677 hybrid_clock.cc:648] HybridClock initialized: now 1781996699613699 us; error 33 us; skew 500 ppm
I20260620 23:04:59.614022 12677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:04:59 dist-test-slave-7g8g krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996699, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:59.616935 12677 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:04:59.618309 12677 webserver.cc:460] Webserver started at http://127.12.2.67:46569/ using document root <none> and password file <none>
I20260620 23:04:59.618968 12677 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:59.619094 12677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:59.619381 12677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:59.622164 12677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "cb1f63792be4421d92e3c641a515c720"
format_stamp: "Formatted at 2026-06-20 23:04:59 on dist-test-slave-7g8g"
server_key: "5e6584f928b6c036397c0ff425251fbf"
server_key_iv: "a33d78c69c8944e04ffb611a03ba7c56"
server_key_version: "encryptionkey@0"
I20260620 23:04:59.622840 12677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "cb1f63792be4421d92e3c641a515c720"
format_stamp: "Formatted at 2026-06-20 23:04:59 on dist-test-slave-7g8g"
server_key: "5e6584f928b6c036397c0ff425251fbf"
server_key_iv: "a33d78c69c8944e04ffb611a03ba7c56"
server_key_version: "encryptionkey@0"
I20260620 23:04:59.627585 12677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:04:59.630849 12692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:59.632134 12677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:04:59.632275 12677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "cb1f63792be4421d92e3c641a515c720"
format_stamp: "Formatted at 2026-06-20 23:04:59 on dist-test-slave-7g8g"
server_key: "5e6584f928b6c036397c0ff425251fbf"
server_key_iv: "a33d78c69c8944e04ffb611a03ba7c56"
server_key_version: "encryptionkey@0"
I20260620 23:04:59.632447 12677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:59.647255 12677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:59.651046 12677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:59.651288 12677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:59.652138 12677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:59.653366 12677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:59.653441 12677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:59.653553 12677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:59.653606 12677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:59.682363 12677 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:33727
I20260620 23:04:59.682405 12804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:33727 every 8 connection(s)
I20260620 23:04:59.683645 12677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:04:59.693909 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 12677
I20260620 23:04:59.694113 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:04:59.694521 12297 external_mini_cluster.cc:1428] Setting key 744faed3029cea1c135625de0f0f3595
Jun 20 23:04:59 dist-test-slave-7g8g krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996699, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:04:59.701671 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:59.686005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:54343 (local address 127.12.2.126:34679)
0620 23:04:59.686167 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:59.686171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:59.687353 (+ 1182us) server_negotiation.cc:407] Connection header received
0620 23:04:59.689039 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:59.689045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:59.689123 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:59.689221 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:59.690618 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.691375 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.692844 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.693255 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.693359 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:59.696679 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:59.696685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:59.696689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:59.696727 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:59.699338 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.699935 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.699939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.699941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.700050 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.700373 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.700377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.700379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.700627 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:59.700727 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:59.701270 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 23:04:59.701427 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:04:59.702579 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:59.686485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34679 (local address 127.12.2.67:54343)
0620 23:04:59.687216 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:59.687253 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:59.688791 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:59.689451 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:59.689459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:59.689928 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:59.690453 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:59.690469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.691512 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.691515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:59.692701 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:59.692716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.693453 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.693460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:59.693699 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:59.694313 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:59.694333 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:59.696466 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:59.699457 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:59.699464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:59.699474 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:59.699849 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:59.700127 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:59.700130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:59.700132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:59.700296 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:59.700714 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:59.700719 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:59.700994 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 23:04:59.701210 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 23:04:59.701548 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":263,"threads_started":1}
I20260620 23:04:59.703831 12805 heartbeater.cc:347] Connected to a master server at 127.12.2.126:34679
I20260620 23:04:59.704062 12805 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:59.704702 12805 heartbeater.cc:511] Master 127.12.2.126:34679 requested a full tablet report, sending...
I20260620 23:04:59.705899 12349 ts_manager.cc:194] Registered new tserver with Master: cb1f63792be4421d92e3c641a515c720 (127.12.2.67:33727)
I20260620 23:04:59.706631 12349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:54343
I20260620 23:04:59.709905 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:04:59 dist-test-slave-7g8g krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996699, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:04:59 dist-test-slave-7g8g krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996699, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:04:59.748816 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:59.735283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36028 (local address 127.12.2.65:36337)
0620 23:04:59.735810 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:59.735817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:59.735966 (+ 149us) server_negotiation.cc:407] Connection header received
0620 23:04:59.736116 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:59.736124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:59.736395 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:59.736569 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:59.737789 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.738750 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.739919 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.740453 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.740553 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:59.744184 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:59.744195 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:59.744208 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:59.744238 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:59.746638 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.747233 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.747240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.747242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.747354 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.747655 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.747658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.747660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.747956 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:59.748170 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:59.748355 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 23:04:59.748545 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":283,"threads_started":1}
W20260620 23:04:59.749996 12453 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36028
Jun 20 23:04:59 dist-test-slave-7g8g krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:04:59 dist-test-slave-7g8g krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:04:59.791846 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:59.779325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.12.2.65:36337)
0620 23:04:59.779523 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:59.779530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:59.779828 (+ 298us) server_negotiation.cc:407] Connection header received
0620 23:04:59.780028 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:59.780032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:59.780104 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:59.780187 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:59.781466 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.782155 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.783379 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.783785 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.783890 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:59.787559 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:59.787564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:59.787567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:59.787598 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:59.789772 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.790333 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.790336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.790339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.790439 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.790813 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.790816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.790818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.791067 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:59.791246 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:59.791481 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 23:04:59.791630 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:04:59.793285 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 12410
I20260620 23:04:59.795554 12538 heartbeater.cc:503] Master 127.12.2.126:34679 was elected leader, sending a full tablet report...
I20260620 23:04:59.805164 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:59.794108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.12.2.65:36337)
0620 23:04:59.794305 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:59.794310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:59.794548 (+ 238us) server_negotiation.cc:407] Connection header received
0620 23:04:59.794673 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:59.794676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:59.794729 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:59.794802 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:59.795880 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.796594 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.797968 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.798380 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.798488 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:59.800624 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:59.800628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:59.800631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:59.800658 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:59.802822 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.803412 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.803418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.803422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.803548 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.803875 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.803888 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.803891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.804126 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:59.804324 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:59.804654 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 23:04:59.804844 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:04:59.805809 12533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:04:59.886004 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 12410
I20260620 23:04:59.899577 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 12544
Jun 20 23:04:59 dist-test-slave-7g8g krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:04:59.914767 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:59.900465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58754 (local address 127.12.2.66:36023)
0620 23:04:59.900952 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:59.900959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:59.900987 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:04:59.901045 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:59.901051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:59.901266 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:59.901401 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:59.902461 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.903534 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.904747 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:59.905327 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:59.905432 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:59.909225 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:59.909236 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:59.909252 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:59.909296 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:59.912047 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.912684 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.912691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.912694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.912831 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:59.913239 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:59.913243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:59.913245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:59.913632 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:59.913869 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:59.914150 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:04:59.914350 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":242,"threads_started":1}
I20260620 23:04:59.916065 12667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:04:59.995275 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 12544
I20260620 23:05:00.011783 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 12677
Jun 20 23:05:00 dist-test-slave-7g8g krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:05:00.026710 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:00.012684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.12.2.67:33727)
0620 23:05:00.013239 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:00.013247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:00.013273 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:05:00.013332 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:00.013338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:00.013548 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:00.013687 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:00.014895 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:00.015923 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:00.017170 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:00.017705 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:00.017822 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:00.021717 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:00.021727 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:00.021743 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:00.021774 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:00.024353 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:00.024939 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:00.024946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:00.024948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:00.025073 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:00.025329 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:00.025332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:00.025334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:00.025649 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:00.025877 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:00.026085 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 23:05:00.026335 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":254,"threads_started":1}
I20260620 23:05:00.028259 12800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:00.108064 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 12677
I20260620 23:05:00.120873 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 12317
I20260620 23:05:00.122009 12379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:00.220681 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 12317
2026-06-20T23:05:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3583 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:00 dist-test-slave-7g8g krb5kdc[12826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:00 dist-test-slave-7g8g krb5kdc[12826](info): set up 2 sockets
Jun 20 23:05:00 dist-test-slave-7g8g krb5kdc[12826](info): commencing operation
krb5kdc: starting...
W20260620 23:05:01.206590 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:01 dist-test-slave-7g8g krb5kdc[12826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:05:01.369817 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:39019
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:01.668993 12839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:01.669293 12839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:01.669428 12839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:01.676430 12839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:01.676541 12839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:01.676616 12839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:01.676702 12839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:01.676759 12839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:01.686738 12839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:39019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:01.688004 12839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:01.689234 12839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:01.700301 12845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:01.700325 12844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:01.700888 12847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:01.700963 12839 server_base.cc:1034] running on GCE node
I20260620 23:05:01.701689 12839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:01.702996 12839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:01.704198 12839 hybrid_clock.cc:648] HybridClock initialized: now 1781996701704145 us; error 43 us; skew 500 ppm
I20260620 23:05:01.704501 12839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:01 dist-test-slave-7g8g krb5kdc[12826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996701, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:01.711743 12839 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:05:01.713207 12839 webserver.cc:460] Webserver started at http://127.12.2.126:44163/ using document root <none> and password file <none>
I20260620 23:05:01.713869 12839 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:01.713980 12839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:01.714269 12839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:01.717197 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "4d5dbeef1f3b493ca734bb66a59ffde3"
format_stamp: "Formatted at 2026-06-20 23:05:01 on dist-test-slave-7g8g"
server_key: "9033d9d68f26ea02fc84f245a386faea"
server_key_iv: "e22fff7e1f401ca9a7a7caa2bc6f289e"
server_key_version: "encryptionkey@0"
I20260620 23:05:01.717895 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "4d5dbeef1f3b493ca734bb66a59ffde3"
format_stamp: "Formatted at 2026-06-20 23:05:01 on dist-test-slave-7g8g"
server_key: "9033d9d68f26ea02fc84f245a386faea"
server_key_iv: "e22fff7e1f401ca9a7a7caa2bc6f289e"
server_key_version: "encryptionkey@0"
I20260620 23:05:01.722808 12839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 23:05:01.726101 12854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:01.727367 12839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:01.727514 12839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "4d5dbeef1f3b493ca734bb66a59ffde3"
format_stamp: "Formatted at 2026-06-20 23:05:01 on dist-test-slave-7g8g"
server_key: "9033d9d68f26ea02fc84f245a386faea"
server_key_iv: "e22fff7e1f401ca9a7a7caa2bc6f289e"
server_key_version: "encryptionkey@0"
I20260620 23:05:01.727687 12839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:01.748762 12839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:01.752542 12839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:01.752794 12839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:01.769182 12839 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:39019
I20260620 23:05:01.769207 12905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:39019 every 8 connection(s)
I20260620 23:05:01.770452 12839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:01.773916 12906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:01.774457 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 12839
I20260620 23:05:01.774657 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:01.775279 12297 external_mini_cluster.cc:1428] Setting key ba19f3fca50cc028d6aed86f89acd0c0
I20260620 23:05:01.786477 12906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3: Bootstrap starting.
Jun 20 23:05:01 dist-test-slave-7g8g krb5kdc[12826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:01.789996 12906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:01.791069 12906 log.cc:824] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:01.793879 12906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3: No bootstrap required, opened a new log
I20260620 23:05:01.797379 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:01.777113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.12.2.126:39019)
0620 23:05:01.777835 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:01.777846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:01.777887 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:05:01.779473 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:01.779494 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:01.779858 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:01.780177 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:01.781826 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:01.783120 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:01.784830 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:01.785477 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:01.785621 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:01.790053 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:01.790061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:01.790076 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:01.790113 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:01.793184 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:01.793937 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:01.793963 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:01.793972 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:01.794156 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:01.794537 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:01.794541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:01.794542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:01.795069 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:01.795268 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:01.795865 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 23:05:01.796275 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":312,"threads_started":1}
I20260620 23:05:01.801534 12906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5dbeef1f3b493ca734bb66a59ffde3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39019 } }
I20260620 23:05:01.801856 12906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:01.801936 12906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5dbeef1f3b493ca734bb66a59ffde3, State: Initialized, Role: FOLLOWER
I20260620 23:05:01.802416 12906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5dbeef1f3b493ca734bb66a59ffde3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39019 } }
I20260620 23:05:01.802618 12906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:01.802727 12906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:01.802866 12906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:01.804859 12906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5dbeef1f3b493ca734bb66a59ffde3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39019 } }
I20260620 23:05:01.805295 12906 leader_election.cc:302] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d5dbeef1f3b493ca734bb66a59ffde3; no voters:
I20260620 23:05:01.805811 12906 leader_election.cc:288] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:01.805958 12911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:01.806895 12911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [term 1 LEADER]: Becoming Leader. State: Replica: 4d5dbeef1f3b493ca734bb66a59ffde3, State: Running, Role: LEADER
I20260620 23:05:01.807406 12911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5dbeef1f3b493ca734bb66a59ffde3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39019 } }
I20260620 23:05:01.807770 12906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:01.810101 12913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d5dbeef1f3b493ca734bb66a59ffde3. Latest consensus state: current_term: 1 leader_uuid: "4d5dbeef1f3b493ca734bb66a59ffde3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5dbeef1f3b493ca734bb66a59ffde3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39019 } } }
I20260620 23:05:01.810463 12913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:01.810979 12912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d5dbeef1f3b493ca734bb66a59ffde3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5dbeef1f3b493ca734bb66a59ffde3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39019 } } }
I20260620 23:05:01.811280 12912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:01.812631 12919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:01.818325 12919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:01.827252 12919 catalog_manager.cc:1347] Generated new cluster ID: 8aa4b9293a704d06baa188b24e34eb15
I20260620 23:05:01.827359 12919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:01.861346 12919 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:01.863152 12919 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:01.877463 12919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d5dbeef1f3b493ca734bb66a59ffde3: Generated new TSK 0
I20260620 23:05:01.878463 12919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:05:01.924887 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39019
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:02.231422 12932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:02.231745 12932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:02.231882 12932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:02.241326 12932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:02.241439 12932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:02.241590 12932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:02.251163 12932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:02.252399 12932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:02.253674 12932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:02.266749 12940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:02.266779 12938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:02.267470 12937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:02.267446 12932 server_base.cc:1034] running on GCE node
I20260620 23:05:02.268090 12932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:02.268967 12932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:02.270167 12932 hybrid_clock.cc:648] HybridClock initialized: now 1781996702270127 us; error 32 us; skew 500 ppm
I20260620 23:05:02.270430 12932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:02 dist-test-slave-7g8g krb5kdc[12826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:02.273329 12932 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:05:02.274721 12932 webserver.cc:460] Webserver started at http://127.12.2.65:38831/ using document root <none> and password file <none>
I20260620 23:05:02.275390 12932 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:02.275497 12932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:02.275774 12932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:02.278563 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "91d8868637d9440094bebdb3c879bf39"
format_stamp: "Formatted at 2026-06-20 23:05:02 on dist-test-slave-7g8g"
server_key: "45fb128614dfd55ab7a37caf28e1c5e2"
server_key_iv: "e0164ff6ffac9001f39a078d1cfef42f"
server_key_version: "encryptionkey@0"
I20260620 23:05:02.279335 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "91d8868637d9440094bebdb3c879bf39"
format_stamp: "Formatted at 2026-06-20 23:05:02 on dist-test-slave-7g8g"
server_key: "45fb128614dfd55ab7a37caf28e1c5e2"
server_key_iv: "e0164ff6ffac9001f39a078d1cfef42f"
server_key_version: "encryptionkey@0"
I20260620 23:05:02.284250 12932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:05:02.287808 12947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:02.289109 12932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:05:02.289264 12932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "91d8868637d9440094bebdb3c879bf39"
format_stamp: "Formatted at 2026-06-20 23:05:02 on dist-test-slave-7g8g"
server_key: "45fb128614dfd55ab7a37caf28e1c5e2"
server_key_iv: "e0164ff6ffac9001f39a078d1cfef42f"
server_key_version: "encryptionkey@0"
I20260620 23:05:02.289441 12932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:02.311287 12932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:02.315599 12932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:02.315867 12932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:02.316846 12932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:02.318110 12932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:02.318192 12932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:02.318310 12932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:02.318361 12932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:02.347795 12932 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:38483
I20260620 23:05:02.347841 13059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:38483 every 8 connection(s)
I20260620 23:05:02.348985 12932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:02.351688 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 12932
I20260620 23:05:02.351904 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:02.352497 12297 external_mini_cluster.cc:1428] Setting key 6fd138ac3ef5ff709d89568502cbefc8
Jun 20 23:05:02 dist-test-slave-7g8g krb5kdc[12826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
I20260620 23:05:02.369534 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:02.351446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:54489 (local address 127.12.2.126:39019)
0620 23:05:02.351929 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:02.351935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:02.352975 (+ 1040us) server_negotiation.cc:407] Connection header received
0620 23:05:02.354778 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:02.354784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:02.354868 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:02.354981 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:02.356439 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:02.357163 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:02.358961 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:02.359366 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:02.359461 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:02.363474 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:02.363479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:02.363482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:02.363516 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:02.366574 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:02.367350 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:02.367356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:02.367360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:02.367500 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:02.367895 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:02.367901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:02.367904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:02.368274 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:02.368417 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:02.369056 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 23:05:02.369227 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260620 23:05:02.370460 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:02.352054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:39019 (local address 127.12.2.65:54489)
0620 23:05:02.352800 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:02.352844 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:02.354518 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:02.355218 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:02.355226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:02.355709 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:02.356269 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:02.356291 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:02.357294 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:02.357298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:02.358783 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:02.358798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:02.359570 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:02.359577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:02.359856 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:02.360535 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:02.360562 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:02.363227 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:02.366729 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:02.366747 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:02.366761 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:02.367233 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:02.367596 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:02.367599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:02.367601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:02.367779 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:02.368377 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:02.368383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:02.368732 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 23:05:02.368954 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 23:05:02.369347 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":285,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
I20260620 23:05:02.371953 13060 heartbeater.cc:347] Connected to a master server at 127.12.2.126:39019
I20260620 23:05:02.372233 13060 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:02.372891 13060 heartbeater.cc:511] Master 127.12.2.126:39019 requested a full tablet report, sending...
I20260620 23:05:02.374919 12871 ts_manager.cc:194] Registered new tserver with Master: 91d8868637d9440094bebdb3c879bf39 (127.12.2.65:38483)
I20260620 23:05:02.376358 12871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:54489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:05:02.389809 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39019
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:02.693550 13066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:02.693823 13066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:02.693954 13066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:02.701048 13066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:02.701161 13066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:02.701313 13066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:02.710897 13066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:02.712110 13066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:02.713410 13066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:02.725538 13071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:02.725565 13074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:02.725600 13072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:02.726774 13066 server_base.cc:1034] running on GCE node
I20260620 23:05:02.727388 13066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:02.728289 13066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:02.729488 13066 hybrid_clock.cc:648] HybridClock initialized: now 1781996702729426 us; error 50 us; skew 500 ppm
I20260620 23:05:02.729828 13066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:02 dist-test-slave-7g8g krb5kdc[12826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:02.732928 13066 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:05:02.734416 13066 webserver.cc:460] Webserver started at http://127.12.2.66:44491/ using document root <none> and password file <none>
I20260620 23:05:02.735085 13066 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:02.735193 13066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:02.735442 13066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:02.738180 13066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "1127fe3e14bf4cd7b7526d304997df2a"
format_stamp: "Formatted at 2026-06-20 23:05:02 on dist-test-slave-7g8g"
server_key: "a8dc21718d77af7dcbc5c2019ee61fc5"
server_key_iv: "37cd524aed3164abf692158ced059371"
server_key_version: "encryptionkey@0"
I20260620 23:05:02.738902 13066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "1127fe3e14bf4cd7b7526d304997df2a"
format_stamp: "Formatted at 2026-06-20 23:05:02 on dist-test-slave-7g8g"
server_key: "a8dc21718d77af7dcbc5c2019ee61fc5"
server_key_iv: "37cd524aed3164abf692158ced059371"
server_key_version: "encryptionkey@0"
I20260620 23:05:02.743877 13066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 23:05:02.747293 13081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:02.748651 13066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:05:02.748802 13066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "1127fe3e14bf4cd7b7526d304997df2a"
format_stamp: "Formatted at 2026-06-20 23:05:02 on dist-test-slave-7g8g"
server_key: "a8dc21718d77af7dcbc5c2019ee61fc5"
server_key_iv: "37cd524aed3164abf692158ced059371"
server_key_version: "encryptionkey@0"
I20260620 23:05:02.748983 13066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:02.760102 13066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:02.763818 13066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:02.764060 13066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:02.764950 13066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:02.766155 13066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:02.766233 13066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:02.766340 13066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:02.766386 13066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:02.796277 13066 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:46567
I20260620 23:05:02.796320 13193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:46567 every 8 connection(s)
I20260620 23:05:02.797489 13066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:02.805048 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 13066
I20260620 23:05:02.805207 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:02.805708 12297 external_mini_cluster.cc:1428] Setting key 82f60b5ba75d8557e1efe82bb4cc35ef
Jun 20 23:05:02 dist-test-slave-7g8g krb5kdc[12826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:02.816190 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:02.799807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:58927 (local address 127.12.2.126:39019)
0620 23:05:02.799977 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:02.799981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:02.801045 (+ 1064us) server_negotiation.cc:407] Connection header received
0620 23:05:02.802753 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:02.802759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:02.802835 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:02.802917 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:02.804360 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:02.805085 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:02.807072 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:02.807618 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:02.807739 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:02.811027 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:02.811032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:02.811035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:02.811063 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:02.813402 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:02.814049 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:02.814055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:02.814058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:02.814192 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:02.814591 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:02.814596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:02.814600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:02.814851 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:02.814994 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:02.815649 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 23:05:02.815855 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:05:02.817185 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:02.800231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:39019 (local address 127.12.2.66:58927)
0620 23:05:02.800895 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:02.800931 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:02.802509 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:02.803162 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:02.803179 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:02.803648 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:02.804205 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:02.804222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:02.805233 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:02.805238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:02.806916 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:02.806963 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:02.807983 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:02.807988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:02.808196 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:02.808796 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:02.808816 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:02.810816 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:02.813523 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:02.813532 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:02.813548 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:02.813944 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:02.814290 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:02.814295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:02.814299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:02.814484 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:02.814985 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:02.814992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:02.815293 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 23:05:02.815527 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:05:02.816017 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":248,"threads_started":1}
I20260620 23:05:02.818722 13194 heartbeater.cc:347] Connected to a master server at 127.12.2.126:39019
I20260620 23:05:02.819056 13194 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:02.819725 13194 heartbeater.cc:511] Master 127.12.2.126:39019 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
I20260620 23:05:02.821099 12871 ts_manager.cc:194] Registered new tserver with Master: 1127fe3e14bf4cd7b7526d304997df2a (127.12.2.66:46567)
I20260620 23:05:02.821866 12871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:58927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:05:02.839584 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39019
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:03.149647 13199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:03.149938 13199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:03.150209 13199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:03.157493 13199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:03.157603 13199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:03.157737 13199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:03.167380 13199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:03.168617 13199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:03.169862 13199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:03.182015 13204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:03.182076 13205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:03.182053 13207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:03.182679 13199 server_base.cc:1034] running on GCE node
I20260620 23:05:03.183290 13199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:03.184170 13199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:03.185362 13199 hybrid_clock.cc:648] HybridClock initialized: now 1781996703185306 us; error 50 us; skew 500 ppm
I20260620 23:05:03.185662 13199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:03 dist-test-slave-7g8g krb5kdc[12826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996703, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:03.188722 13199 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:05:03.190299 13199 webserver.cc:460] Webserver started at http://127.12.2.67:44305/ using document root <none> and password file <none>
I20260620 23:05:03.191051 13199 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:03.191184 13199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:03.191490 13199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:03.194591 13199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "3843180f05194061a94faee529c99940"
format_stamp: "Formatted at 2026-06-20 23:05:03 on dist-test-slave-7g8g"
server_key: "13d6bec30d79ef5a40285fce468e92f8"
server_key_iv: "7866431b8f6825ee83cc70c830ce86f2"
server_key_version: "encryptionkey@0"
I20260620 23:05:03.195400 13199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "3843180f05194061a94faee529c99940"
format_stamp: "Formatted at 2026-06-20 23:05:03 on dist-test-slave-7g8g"
server_key: "13d6bec30d79ef5a40285fce468e92f8"
server_key_iv: "7866431b8f6825ee83cc70c830ce86f2"
server_key_version: "encryptionkey@0"
I20260620 23:05:03.200466 13199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:05:03.204006 13214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:03.205302 13199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:05:03.205456 13199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "3843180f05194061a94faee529c99940"
format_stamp: "Formatted at 2026-06-20 23:05:03 on dist-test-slave-7g8g"
server_key: "13d6bec30d79ef5a40285fce468e92f8"
server_key_iv: "7866431b8f6825ee83cc70c830ce86f2"
server_key_version: "encryptionkey@0"
I20260620 23:05:03.205641 13199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:03.228731 13199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:03.233198 13199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:03.233481 13199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:03.234467 13199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:03.235899 13199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:03.235988 13199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:03.236115 13199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:03.236182 13199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:03.267180 13199 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:34289
I20260620 23:05:03.267251 13326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:34289 every 8 connection(s)
I20260620 23:05:03.268433 13199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:03.274072 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 13199
I20260620 23:05:03.274286 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:03.274703 12297 external_mini_cluster.cc:1428] Setting key 39fc94e92753c5706a0275e46ca4b8d2
Jun 20 23:05:03 dist-test-slave-7g8g krb5kdc[12826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996703, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:03.288370 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.270876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:35175 (local address 127.12.2.126:39019)
0620 23:05:03.271094 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.271100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.272203 (+ 1103us) server_negotiation.cc:407] Connection header received
0620 23:05:03.273973 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.273980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.274065 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.274180 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:03.275800 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.276631 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.278462 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.278910 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.279052 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.282609 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:03.282614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:03.282618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:03.282652 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:03.285222 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.285969 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.285974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.285976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.286117 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.286516 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.286521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.286523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.286878 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:03.287056 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.287758 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.287987 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:05:03.289273 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.271382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:39019 (local address 127.12.2.67:35175)
0620 23:05:03.272051 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:03.272087 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:03.273699 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:03.274439 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:03.274448 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:03.275047 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:03.275619 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.275636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.276780 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.276785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.278328 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.278341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.279176 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.279182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.279418 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.280093 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:03.280119 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:03.282393 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:03.285359 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:03.285366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:03.285380 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:03.285829 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:03.286212 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:03.286216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:03.286218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:03.286397 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:03.287041 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:03.287048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:03.287392 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 23:05:03.287626 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:05:03.288031 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":253,"threads_started":1}
I20260620 23:05:03.290548 13327 heartbeater.cc:347] Connected to a master server at 127.12.2.126:39019
I20260620 23:05:03.290864 13327 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:03.291536 13327 heartbeater.cc:511] Master 127.12.2.126:39019 requested a full tablet report, sending...
I20260620 23:05:03.292873 12870 ts_manager.cc:194] Registered new tserver with Master: 3843180f05194061a94faee529c99940 (127.12.2.67:34289)
I20260620 23:05:03.293692 12870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:35175
I20260620 23:05:03.302438 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:05:03.320487 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.308670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.12.2.126:39019)
0620 23:05:03.308833 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.308837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.309373 (+ 536us) server_negotiation.cc:407] Connection header received
0620 23:05:03.309563 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.309567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.309628 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.309703 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:03.311230 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.311930 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.313156 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.313561 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.313647 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.315946 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:03.315951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:03.315954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:03.315984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:03.318381 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.319023 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.319027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.319029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.319144 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.319425 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.319428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.319430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.319704 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:03.319877 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.320120 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.320266 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:05:03.326704 12870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:03.333261 12870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:03.367242 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.344816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:46567 (local address 127.0.0.1:52282)
0620 23:05:03.352139 (+ 7323us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:03.352184 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:03.352383 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:03.361438 (+ 9055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:03.361442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:03.361465 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:03.361787 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.361796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.363135 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.363138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.364790 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.364803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.366234 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.366240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.366417 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.366458 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:05:03.366581 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 23:05:03.366761 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7028,"mutex_wait_us":32,"thread_start_us":6851,"threads_started":1}
I20260620 23:05:03.367457 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.352659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.12.2.65:38483)
0620 23:05:03.358497 (+ 5838us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.358507 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.359256 (+ 749us) server_negotiation.cc:407] Connection header received
0620 23:05:03.359431 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.359438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.359668 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.360599 (+ 931us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:03.360826 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.362128 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.365058 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.366432 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.366549 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.366758 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.366998 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.367162 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5649,"thread_start_us":5411,"threads_started":1}
I20260620 23:05:03.367710 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.345018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:34289 (local address 127.0.0.1:54734)
0620 23:05:03.356156 (+ 11138us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:03.356177 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:03.356309 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:03.360037 (+ 3728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:03.360046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:03.360074 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:03.360554 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.360565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.362158 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.362161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.364899 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.364909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.366906 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.366912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.367108 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.367139 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:05:03.367227 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:05:03.367366 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10961,"mutex_wait_us":188,"thread_start_us":11536,"threads_started":1}
I20260620 23:05:03.367895 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.345012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.12.2.66:46567)
0620 23:05:03.360895 (+ 15883us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.360904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.360928 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:05:03.361017 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.361024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.361248 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.361427 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:03.361938 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.363018 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.364970 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.366051 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.366162 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.366356 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.367468 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.367629 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15730,"thread_start_us":15638,"threads_started":1}
I20260620 23:05:03.368055 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.357652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38483 (local address 127.0.0.1:52868)
0620 23:05:03.359153 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:03.359168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:03.359285 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:03.359908 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:03.359915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:03.359944 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:03.360396 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.360409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.362249 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.362254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.364881 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.364899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.366612 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.366619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.366791 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.366819 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:05:03.366920 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:05:03.367060 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1404,"mutex_wait_us":175,"thread_start_us":1288,"threads_started":1}
I20260620 23:05:03.368521 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.345664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.12.2.67:34289)
0620 23:05:03.359088 (+ 13424us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.359097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.359135 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:05:03.359255 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.359264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.359535 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.359735 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:03.360767 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.362038 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.365058 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.366426 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.366550 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.366758 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.368051 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.368206 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13229,"thread_start_us":13697,"threads_started":1}
I20260620 23:05:03.371469 13129 tablet_service.cc:1467] Processing CreateTablet for tablet d252537b93594f8985cabb6079540580 (DEFAULT_TABLE table=test-table [id=76d35b0c3b0449f9b66190520fbdc3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:03.371517 12995 tablet_service.cc:1467] Processing CreateTablet for tablet d252537b93594f8985cabb6079540580 (DEFAULT_TABLE table=test-table [id=76d35b0c3b0449f9b66190520fbdc3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:03.371519 13262 tablet_service.cc:1467] Processing CreateTablet for tablet d252537b93594f8985cabb6079540580 (DEFAULT_TABLE table=test-table [id=76d35b0c3b0449f9b66190520fbdc3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:03.372601 13129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d252537b93594f8985cabb6079540580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:03.372740 12995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d252537b93594f8985cabb6079540580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:03.372763 13262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d252537b93594f8985cabb6079540580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:03.379122 13060 heartbeater.cc:503] Master 127.12.2.126:39019 was elected leader, sending a full tablet report...
I20260620 23:05:03.383562 13345 tablet_bootstrap.cc:492] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a: Bootstrap starting.
I20260620 23:05:03.385975 13346 tablet_bootstrap.cc:492] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39: Bootstrap starting.
I20260620 23:05:03.386703 13347 tablet_bootstrap.cc:492] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940: Bootstrap starting.
I20260620 23:05:03.387672 13345 tablet_bootstrap.cc:654] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:03.388612 13345 log.cc:824] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:03.390019 13346 tablet_bootstrap.cc:654] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:03.390019 13347 tablet_bootstrap.cc:654] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:03.391086 13345 tablet_bootstrap.cc:492] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a: No bootstrap required, opened a new log
I20260620 23:05:03.391191 13347 log.cc:824] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:03.391181 13346 log.cc:824] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:03.391330 13345 ts_tablet_manager.cc:1397] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 23:05:03.393839 13347 tablet_bootstrap.cc:492] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940: No bootstrap required, opened a new log
I20260620 23:05:03.394069 13347 ts_tablet_manager.cc:1397] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 23:05:03.394634 13346 tablet_bootstrap.cc:492] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39: No bootstrap required, opened a new log
I20260620 23:05:03.394891 13346 ts_tablet_manager.cc:1397] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 23:05:03.401178 13347 raft_consensus.cc:348] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127fe3e14bf4cd7b7526d304997df2a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46567 } } peers { permanent_uuid: "3843180f05194061a94faee529c99940" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34289 } } peers { permanent_uuid: "91d8868637d9440094bebdb3c879bf39" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38483 } }
I20260620 23:05:03.401515 13347 raft_consensus.cc:374] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:03.401243 13345 raft_consensus.cc:348] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127fe3e14bf4cd7b7526d304997df2a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46567 } } peers { permanent_uuid: "3843180f05194061a94faee529c99940" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34289 } } peers { permanent_uuid: "91d8868637d9440094bebdb3c879bf39" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38483 } }
I20260620 23:05:03.401618 13347 raft_consensus.cc:729] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3843180f05194061a94faee529c99940, State: Initialized, Role: FOLLOWER
I20260620 23:05:03.401633 13345 raft_consensus.cc:374] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:03.401710 13345 raft_consensus.cc:729] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1127fe3e14bf4cd7b7526d304997df2a, State: Initialized, Role: FOLLOWER
I20260620 23:05:03.402236 13347 consensus_queue.cc:260] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127fe3e14bf4cd7b7526d304997df2a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46567 } } peers { permanent_uuid: "3843180f05194061a94faee529c99940" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34289 } } peers { permanent_uuid: "91d8868637d9440094bebdb3c879bf39" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38483 } }
I20260620 23:05:03.402424 13345 consensus_queue.cc:260] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127fe3e14bf4cd7b7526d304997df2a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46567 } } peers { permanent_uuid: "3843180f05194061a94faee529c99940" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34289 } } peers { permanent_uuid: "91d8868637d9440094bebdb3c879bf39" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38483 } }
I20260620 23:05:03.403246 13327 heartbeater.cc:503] Master 127.12.2.126:39019 was elected leader, sending a full tablet report...
I20260620 23:05:03.403944 13347 ts_tablet_manager.cc:1428] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260620 23:05:03.405193 13194 heartbeater.cc:503] Master 127.12.2.126:39019 was elected leader, sending a full tablet report...
I20260620 23:05:03.405042 13346 raft_consensus.cc:348] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127fe3e14bf4cd7b7526d304997df2a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46567 } } peers { permanent_uuid: "3843180f05194061a94faee529c99940" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34289 } } peers { permanent_uuid: "91d8868637d9440094bebdb3c879bf39" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38483 } }
I20260620 23:05:03.405421 13346 raft_consensus.cc:374] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:03.405515 13346 raft_consensus.cc:729] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d8868637d9440094bebdb3c879bf39, State: Initialized, Role: FOLLOWER
I20260620 23:05:03.405879 13345 ts_tablet_manager.cc:1428] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 23:05:03.406073 13346 consensus_queue.cc:260] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127fe3e14bf4cd7b7526d304997df2a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46567 } } peers { permanent_uuid: "3843180f05194061a94faee529c99940" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34289 } } peers { permanent_uuid: "91d8868637d9440094bebdb3c879bf39" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38483 } }
I20260620 23:05:03.407734 13346 ts_tablet_manager.cc:1428] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260620 23:05:03.409874 13354 raft_consensus.cc:482] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:03.410212 13354 raft_consensus.cc:504] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127fe3e14bf4cd7b7526d304997df2a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46567 } } peers { permanent_uuid: "3843180f05194061a94faee529c99940" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34289 } } peers { permanent_uuid: "91d8868637d9440094bebdb3c879bf39" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38483 } }
I20260620 23:05:03.411949 13354 leader_election.cc:288] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1127fe3e14bf4cd7b7526d304997df2a (127.12.2.66:46567), 3843180f05194061a94faee529c99940 (127.12.2.67:34289)
I20260620 23:05:03.418906 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.412703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:39051 (local address 127.12.2.66:46567)
0620 23:05:03.412851 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.412855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.412926 (+ 71us) server_negotiation.cc:407] Connection header received
0620 23:05:03.413105 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.413109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.413160 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.413233 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:03.413840 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.414556 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.417158 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.418002 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.418121 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.418199 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.418599 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.418727 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:05:03.419060 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.412260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:46567 (local address 127.12.2.65:39051)
0620 23:05:03.412826 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:03.412844 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:03.412990 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:03.413258 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:03.413263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:03.413285 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:03.413661 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.413673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.414662 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.414667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.416972 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.416990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.418218 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.418225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.418417 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.418455 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:05:03.418550 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 23:05:03.418682 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"threads_started":1}
I20260620 23:05:03.420425 13149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d252537b93594f8985cabb6079540580" candidate_uuid: "91d8868637d9440094bebdb3c879bf39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127fe3e14bf4cd7b7526d304997df2a" is_pre_election: true
I20260620 23:05:03.420897 13149 raft_consensus.cc:2454] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d8868637d9440094bebdb3c879bf39 in term 0.
I20260620 23:05:03.421592 12949 leader_election.cc:302] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1127fe3e14bf4cd7b7526d304997df2a, 91d8868637d9440094bebdb3c879bf39; no voters:
I20260620 23:05:03.422003 13354 raft_consensus.cc:2781] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:03.422174 13354 raft_consensus.cc:482] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:03.422281 13354 raft_consensus.cc:3037] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:03.424724 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.412287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:34289 (local address 127.12.2.65:34133)
0620 23:05:03.419290 (+ 7003us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:03.419312 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:03.419431 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:03.419736 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:03.419741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:03.419765 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:03.419898 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.419904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.420856 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.420861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.422478 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:03.422490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.424137 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.424142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:03.424315 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.424339 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:05:03.424420 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:05:03.424525 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6885,"thread_start_us":568,"threads_started":1}
I20260620 23:05:03.424847 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.412675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:34133 (local address 127.12.2.67:34289)
0620 23:05:03.412843 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.412848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.419396 (+ 6548us) server_negotiation.cc:407] Connection header received
0620 23:05:03.419544 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.419549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.419624 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.419708 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:03.420004 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.420733 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.422682 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.423978 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.424119 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.424291 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.424461 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.424587 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:05:03.425595 13354 raft_consensus.cc:504] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127fe3e14bf4cd7b7526d304997df2a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46567 } } peers { permanent_uuid: "3843180f05194061a94faee529c99940" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34289 } } peers { permanent_uuid: "91d8868637d9440094bebdb3c879bf39" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38483 } }
I20260620 23:05:03.425767 13282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d252537b93594f8985cabb6079540580" candidate_uuid: "91d8868637d9440094bebdb3c879bf39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3843180f05194061a94faee529c99940" is_pre_election: true
I20260620 23:05:03.426276 13354 leader_election.cc:288] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [CANDIDATE]: Term 1 election: Requested vote from peers 1127fe3e14bf4cd7b7526d304997df2a (127.12.2.66:46567), 3843180f05194061a94faee529c99940 (127.12.2.67:34289)
I20260620 23:05:03.426261 13282 raft_consensus.cc:2454] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d8868637d9440094bebdb3c879bf39 in term 0.
I20260620 23:05:03.426597 13149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d252537b93594f8985cabb6079540580" candidate_uuid: "91d8868637d9440094bebdb3c879bf39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127fe3e14bf4cd7b7526d304997df2a"
I20260620 23:05:03.426803 13149 raft_consensus.cc:3037] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:03.427014 13282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d252537b93594f8985cabb6079540580" candidate_uuid: "91d8868637d9440094bebdb3c879bf39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3843180f05194061a94faee529c99940"
I20260620 23:05:03.427188 13282 raft_consensus.cc:3037] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:03.428938 13149 raft_consensus.cc:2454] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d8868637d9440094bebdb3c879bf39 in term 1.
I20260620 23:05:03.429208 13282 raft_consensus.cc:2454] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d8868637d9440094bebdb3c879bf39 in term 1.
I20260620 23:05:03.429235 12949 leader_election.cc:302] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1127fe3e14bf4cd7b7526d304997df2a, 91d8868637d9440094bebdb3c879bf39; no voters:
I20260620 23:05:03.429569 13354 raft_consensus.cc:2781] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:03.429703 13354 raft_consensus.cc:686] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [term 1 LEADER]: Becoming Leader. State: Replica: 91d8868637d9440094bebdb3c879bf39, State: Running, Role: LEADER
I20260620 23:05:03.430164 13354 consensus_queue.cc:237] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127fe3e14bf4cd7b7526d304997df2a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46567 } } peers { permanent_uuid: "3843180f05194061a94faee529c99940" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34289 } } peers { permanent_uuid: "91d8868637d9440094bebdb3c879bf39" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38483 } }
I20260620 23:05:03.434239 12870 catalog_manager.cc:5495] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d8868637d9440094bebdb3c879bf39 (127.12.2.65). New cstate: current_term: 1 leader_uuid: "91d8868637d9440094bebdb3c879bf39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127fe3e14bf4cd7b7526d304997df2a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3843180f05194061a94faee529c99940" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d8868637d9440094bebdb3c879bf39" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38483 } health_report { overall_health: HEALTHY } } }
Jun 20 23:05:03 dist-test-slave-7g8g krb5kdc[12826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:05:03.461426 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.448210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.12.2.65:38483)
0620 23:05:03.448380 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.448384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.448844 (+ 460us) server_negotiation.cc:407] Connection header received
0620 23:05:03.448992 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.448996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.449062 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.449142 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:03.450442 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.451222 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.452429 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.452865 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.452961 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.456617 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:03.456648 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:03.456663 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:03.456694 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:03.459362 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.459917 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.459924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.459927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.460051 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.460319 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.460322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.460324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.460632 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:03.460833 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.461041 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.461191 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 23:05:03 dist-test-slave-7g8g krb5kdc[12826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996703, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:05:03 dist-test-slave-7g8g krb5kdc[12826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996703, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:05:03.502419 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.489772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.12.2.65:38483)
0620 23:05:03.489953 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.489959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.490327 (+ 368us) server_negotiation.cc:407] Connection header received
0620 23:05:03.490498 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.490502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.490562 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.490643 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:03.491932 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.492647 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.493945 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.494393 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.494508 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.497946 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:03.497950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:03.497954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:03.497985 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:03.500351 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.500983 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.500987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.500989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.501112 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.501413 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.501416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.501418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.501635 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:03.501827 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.502019 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.502168 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 23:05:03.519964 13328 tablet.cc:2369] T d252537b93594f8985cabb6079540580 P 3843180f05194061a94faee529c99940: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 23:05:03 dist-test-slave-7g8g krb5kdc[12826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:05:03 dist-test-slave-7g8g krb5kdc[12826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:05:03.542460 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.529457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52912 (local address 127.12.2.65:38483)
0620 23:05:03.529653 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.529660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.529929 (+ 269us) server_negotiation.cc:407] Connection header received
0620 23:05:03.530093 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.530098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.530179 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.530272 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:03.531565 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.532436 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.533754 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.534186 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.534301 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.537831 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:03.537836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:03.537839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:03.537872 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:03.540122 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.540726 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.540730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.540733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.540863 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.541186 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.541190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.541192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.541433 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:03.541668 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.541935 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.542139 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 23:05:03.544140 12975 tablet_service.cc:2908] Rejecting scan request for tablet d252537b93594f8985cabb6079540580: Uninitialized: clean time has not yet been initialized
I20260620 23:05:03.546761 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 12932
W20260620 23:05:03.549257 13195 tablet.cc:2369] T d252537b93594f8985cabb6079540580 P 1127fe3e14bf4cd7b7526d304997df2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:03.558401 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.547575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.12.2.65:38483)
0620 23:05:03.547709 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.547713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.548029 (+ 316us) server_negotiation.cc:407] Connection header received
0620 23:05:03.548139 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.548143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.548193 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.548264 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:03.549442 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.550182 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.551408 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.551773 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.551908 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.553889 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:03.553895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:03.553898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:03.553924 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:03.556150 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.556741 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.556746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.556750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.556861 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.557166 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.557171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.557175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.557417 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:03.557642 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.557905 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.558091 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:05:03.559054 13055 generic_service.cc:196] Checking for leaks (request via RPC)
W20260620 23:05:03.645285 13061 tablet.cc:2369] T d252537b93594f8985cabb6079540580 P 91d8868637d9440094bebdb3c879bf39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:03.645893 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 12932
I20260620 23:05:03.659029 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 13066
Jun 20 23:05:03 dist-test-slave-7g8g krb5kdc[12826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:05:03.672293 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.659684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.12.2.66:46567)
0620 23:05:03.659878 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.659884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.659904 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:05:03.660049 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.660053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.660129 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.660238 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:03.661298 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.662062 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.663308 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.663749 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.663854 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.667603 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:03.667615 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:03.667630 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:03.667663 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:03.670175 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.670716 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.670722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.670724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.670835 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.671138 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.671141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.671143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.671492 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:03.671716 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.671903 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.672048 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:05:03.672861 13189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:03.756170 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 13066
I20260620 23:05:03.768319 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 13199
Jun 20 23:05:03 dist-test-slave-7g8g krb5kdc[12826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:05:03.783532 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:03.769038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.12.2.67:34289)
0620 23:05:03.769232 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:03.769238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:03.769258 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:05:03.769319 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:03.769324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:03.769397 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:03.769507 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:03.770591 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.771700 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.772965 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:03.773591 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:03.773705 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:03.777453 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:03.777471 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:03.777491 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:03.777531 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:03.780989 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.781581 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.781590 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.781593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.781744 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:03.782099 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:03.782104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:03.782108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:03.782511 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:03.782807 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:03.782988 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 23:05:03.783188 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:05:03.784121 13322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:03.869091 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 13199
I20260620 23:05:03.881186 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 12839
I20260620 23:05:03.882230 12901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:03.987447 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 12839
2026-06-20T23:05:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3767 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:04 dist-test-slave-7g8g krb5kdc[13375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:04 dist-test-slave-7g8g krb5kdc[13375](info): set up 2 sockets
Jun 20 23:05:04 dist-test-slave-7g8g krb5kdc[13375](info): commencing operation
krb5kdc: starting...
W20260620 23:05:04.974828 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:05 dist-test-slave-7g8g krb5kdc[13375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:05:05.139539 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43961
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:43961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:05.443403 13388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:05.443689 13388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:05.443805 13388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:05.450812 13388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:05.450918 13388 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:05:05.451001 13388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:05.451082 13388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:05.451148 13388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:05.451227 13388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:05.460813 13388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:43961
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:05.462056 13388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:05.463380 13388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:05.474263 13393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:05.474220 13396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:05.474283 13394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:05.474910 13388 server_base.cc:1034] running on GCE node
I20260620 23:05:05.475597 13388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:05.476809 13388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:05.478000 13388 hybrid_clock.cc:648] HybridClock initialized: now 1781996705477939 us; error 40 us; skew 500 ppm
I20260620 23:05:05.478263 13388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:05 dist-test-slave-7g8g krb5kdc[13375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996705, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:05.480997 13388 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:05:05.482302 13388 webserver.cc:460] Webserver started at http://127.12.2.126:44099/ using document root <none> and password file <none>
I20260620 23:05:05.482916 13388 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:05.483033 13388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:05.483290 13388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:05.486064 13388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "fde20c95a9974a8bbc2fa7af089f19ec"
format_stamp: "Formatted at 2026-06-20 23:05:05 on dist-test-slave-7g8g"
server_key: "0184b97892a66b78eeb6f037344aa7b5"
server_key_iv: "42b63240342372890d520f9238fa1f85"
server_key_version: "encryptionkey@0"
I20260620 23:05:05.486758 13388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "fde20c95a9974a8bbc2fa7af089f19ec"
format_stamp: "Formatted at 2026-06-20 23:05:05 on dist-test-slave-7g8g"
server_key: "0184b97892a66b78eeb6f037344aa7b5"
server_key_iv: "42b63240342372890d520f9238fa1f85"
server_key_version: "encryptionkey@0"
I20260620 23:05:05.491430 13388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:05:05.494524 13403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:05.495815 13388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:05.495961 13388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "fde20c95a9974a8bbc2fa7af089f19ec"
format_stamp: "Formatted at 2026-06-20 23:05:05 on dist-test-slave-7g8g"
server_key: "0184b97892a66b78eeb6f037344aa7b5"
server_key_iv: "42b63240342372890d520f9238fa1f85"
server_key_version: "encryptionkey@0"
I20260620 23:05:05.496142 13388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:05.513084 13388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:05.516798 13388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:05.517016 13388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:05.536046 13388 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:43961
I20260620 23:05:05.536072 13464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:43961 every 8 connection(s)
I20260620 23:05:05.537302 13388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:05.540992 13465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:05.544207 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 13388
I20260620 23:05:05.544440 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:05.545181 12297 external_mini_cluster.cc:1428] Setting key 2bae9352b88c4152c49cda1d1e608d9f
I20260620 23:05:05.553424 13465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec: Bootstrap starting.
I20260620 23:05:05.557055 13465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:05.558282 13465 log.cc:824] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:05:05 dist-test-slave-7g8g krb5kdc[13375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:05.561347 13465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec: No bootstrap required, opened a new log
I20260620 23:05:05.566962 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:05.547201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.12.2.126:43961)
0620 23:05:05.547896 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:05.547907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:05.547946 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:05:05.549548 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:05.549570 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:05.550014 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:05.550346 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:05.551959 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:05.553202 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:05.554856 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:05.555574 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:05.555696 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:05.560241 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:05.560249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:05.560263 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:05.560295 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:05.563196 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:05.563967 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:05.563973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:05.563978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:05.564131 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:05.564476 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:05.564479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:05.564481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:05.564975 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:05.565183 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:05.565587 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 23:05:05.565933 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":316,"threads_started":1}
I20260620 23:05:05.570039 13465 raft_consensus.cc:348] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde20c95a9974a8bbc2fa7af089f19ec" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43961 } }
I20260620 23:05:05.570359 13465 raft_consensus.cc:374] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:05.570443 13465 raft_consensus.cc:729] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde20c95a9974a8bbc2fa7af089f19ec, State: Initialized, Role: FOLLOWER
I20260620 23:05:05.570988 13465 consensus_queue.cc:260] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde20c95a9974a8bbc2fa7af089f19ec" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43961 } }
I20260620 23:05:05.571205 13465 raft_consensus.cc:388] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:05.571326 13465 raft_consensus.cc:482] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:05.571467 13465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:05.573457 13465 raft_consensus.cc:504] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde20c95a9974a8bbc2fa7af089f19ec" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43961 } }
I20260620 23:05:05.573974 13465 leader_election.cc:302] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fde20c95a9974a8bbc2fa7af089f19ec; no voters:
I20260620 23:05:05.574452 13465 leader_election.cc:288] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:05.574549 13470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:05.575170 13470 raft_consensus.cc:686] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [term 1 LEADER]: Becoming Leader. State: Replica: fde20c95a9974a8bbc2fa7af089f19ec, State: Running, Role: LEADER
I20260620 23:05:05.575598 13470 consensus_queue.cc:237] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde20c95a9974a8bbc2fa7af089f19ec" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43961 } }
I20260620 23:05:05.576328 13465 sys_catalog.cc:564] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:05.578058 13472 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader fde20c95a9974a8bbc2fa7af089f19ec. Latest consensus state: current_term: 1 leader_uuid: "fde20c95a9974a8bbc2fa7af089f19ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde20c95a9974a8bbc2fa7af089f19ec" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43961 } } }
I20260620 23:05:05.578038 13471 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fde20c95a9974a8bbc2fa7af089f19ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde20c95a9974a8bbc2fa7af089f19ec" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43961 } } }
I20260620 23:05:05.578496 13472 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:05.578593 13471 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:05.579329 13476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:05.583611 13476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:05.590713 13476 catalog_manager.cc:1347] Generated new cluster ID: a974611ce24b4d56a4debd1dd514ebed
I20260620 23:05:05.590807 13476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:05.609735 13476 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:05.611172 13476 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:05.620451 13476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec: Generated new TSK 0
I20260620 23:05:05.621213 13476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:05:05.674048 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43961
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:05.994748 13491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:05.995083 13491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:05.995258 13491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:06.002549 13491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:06.002698 13491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:06.002804 13491 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:05:06.002897 13491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:06.012792 13491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.2.126:43961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:06.014079 13491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:06.015470 13491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:06.027915 13496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:06.027887 13499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:06.027854 13497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:06.028566 13491 server_base.cc:1034] running on GCE node
I20260620 23:05:06.029237 13491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:06.030118 13491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:06.031323 13491 hybrid_clock.cc:648] HybridClock initialized: now 1781996706031251 us; error 51 us; skew 500 ppm
I20260620 23:05:06.031605 13491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:06 dist-test-slave-7g8g krb5kdc[13375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996706, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:06.034632 13491 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:05:06.035981 13491 webserver.cc:460] Webserver started at http://127.12.2.65:43299/ using document root <none> and password file <none>
I20260620 23:05:06.036628 13491 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:06.036712 13491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:06.036981 13491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:06.039783 13491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "37acc82fa373490580f86b01fa0d3907"
format_stamp: "Formatted at 2026-06-20 23:05:06 on dist-test-slave-7g8g"
server_key: "46489c3e33fdb1efb104382b27ccccb6"
server_key_iv: "b8b5d7982d0b6d2cb37b35b8e0c8bf0a"
server_key_version: "encryptionkey@0"
I20260620 23:05:06.040490 13491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "37acc82fa373490580f86b01fa0d3907"
format_stamp: "Formatted at 2026-06-20 23:05:06 on dist-test-slave-7g8g"
server_key: "46489c3e33fdb1efb104382b27ccccb6"
server_key_iv: "b8b5d7982d0b6d2cb37b35b8e0c8bf0a"
server_key_version: "encryptionkey@0"
I20260620 23:05:06.045192 13491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260620 23:05:06.048914 13506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:06.050238 13491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:05:06.050396 13491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "37acc82fa373490580f86b01fa0d3907"
format_stamp: "Formatted at 2026-06-20 23:05:06 on dist-test-slave-7g8g"
server_key: "46489c3e33fdb1efb104382b27ccccb6"
server_key_iv: "b8b5d7982d0b6d2cb37b35b8e0c8bf0a"
server_key_version: "encryptionkey@0"
I20260620 23:05:06.050585 13491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:06.075291 13491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:06.079721 13491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:06.079977 13491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:06.082525 13491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:06.082625 13491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:06.082742 13491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:06.082808 13491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:05:06 dist-test-slave-7g8g krb5kdc[13375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996706, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:06.105723 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:06.083156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:58475 (local address 127.12.2.126:43961)
0620 23:05:06.085516 (+ 2360us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:06.085523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:06.085547 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:05:06.087372 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:06.087377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:06.087443 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:06.087535 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:06.089618 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.090546 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.093430 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.093949 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.094080 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:06.098600 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:06.098605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:06.098609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:06.098641 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:06.101940 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:06.102970 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:06.102976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:06.102980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:06.103127 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:06.103747 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:06.103753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:06.103757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:06.104141 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:06.104292 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:06.105065 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 23:05:06.105313 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2240,"thread_start_us":2139,"threads_started":1}
I20260620 23:05:06.106597 13516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:06.083873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43961 (local address 127.12.2.65:58475)
0620 23:05:06.084800 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:06.084863 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:06.086918 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:06.087832 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:06.087845 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:06.088584 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:06.089442 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:06.089471 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.090718 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.090723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:06.093260 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:06.093277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.094206 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.094215 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:06.094542 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:06.095411 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:06.095442 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:06.098388 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:06.102093 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:06.102104 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:06.102123 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:06.102775 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:06.103256 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:06.103261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:06.103265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:06.103577 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:06.104293 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:06.104301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:06.104696 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 23:05:06.104931 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 23:05:06.105320 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":353,"threads_started":1}
I20260620 23:05:06.124683 13491 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:37861
I20260620 23:05:06.124735 13626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:37861 every 8 connection(s)
I20260620 23:05:06.126118 13491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:06.127234 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:06.113545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:36129 (local address 127.12.2.126:43961)
0620 23:05:06.113693 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:06.113699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:06.113979 (+ 280us) server_negotiation.cc:407] Connection header received
0620 23:05:06.114146 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:06.114149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:06.114208 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:06.114300 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:06.115348 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.116205 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.118195 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.118699 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.118815 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:06.121185 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:06.121192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:06.121197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:06.121236 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:06.124430 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:06.125192 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:06.125198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:06.125201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:06.125366 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:06.125733 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:06.125738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:06.125741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:06.126072 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:06.126210 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:06.126616 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 23:05:06.126833 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:05:06.127286 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:06.113160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43961 (local address 127.12.2.65:36129)
0620 23:05:06.113871 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:06.113898 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:06.114041 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:06.114344 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:06.114352 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:06.114855 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:06.115209 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:06.115224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.116357 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.116363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:06.118059 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:06.118074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.119061 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.119068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:06.119267 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:06.119968 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:06.119993 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:06.120982 (+ 989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:06.124563 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:06.124568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:06.124572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:06.125075 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:06.125497 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:06.125501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:06.125503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:06.125627 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:06.126189 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:06.126194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:06.126371 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 23:05:06.126484 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:05:06.126811 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":430,"threads_started":1}
I20260620 23:05:06.128347 13627 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43961
I20260620 23:05:06.128618 13627 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:06.129432 13627 heartbeater.cc:511] Master 127.12.2.126:43961 requested a full tablet report, sending...
I20260620 23:05:06.131525 13420 ts_manager.cc:194] Registered new tserver with Master: 37acc82fa373490580f86b01fa0d3907 (127.12.2.65:37861)
I20260620 23:05:06.132716 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 13491
I20260620 23:05:06.132920 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:06.133245 13420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:58475
I20260620 23:05:06.133644 12297 external_mini_cluster.cc:1428] Setting key 6c62b61419d79bc59b2e12010de6e69c
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:05:06.169770 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43961
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:06.471947 13631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:06.472232 13631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:06.472353 13631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:06.479476 13631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:06.479588 13631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:06.479708 13631 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:05:06.479815 13631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:06.489374 13631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.2.126:43961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:06.490641 13631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:06.491994 13631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:06.504490 13637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:06.504545 13636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:06.504546 13639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:06.505422 13631 server_base.cc:1034] running on GCE node
I20260620 23:05:06.506078 13631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:06.507077 13631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:06.508317 13631 hybrid_clock.cc:648] HybridClock initialized: now 1781996706508234 us; error 53 us; skew 500 ppm
I20260620 23:05:06.508632 13631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:06 dist-test-slave-7g8g krb5kdc[13375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996706, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:06.511735 13631 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:05:06.513075 13631 webserver.cc:460] Webserver started at http://127.12.2.66:41385/ using document root <none> and password file <none>
I20260620 23:05:06.513788 13631 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:06.513873 13631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:06.514125 13631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:06.516923 13631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "4f3c252d77e44e11b14089434aa5099a"
format_stamp: "Formatted at 2026-06-20 23:05:06 on dist-test-slave-7g8g"
server_key: "b6045a0837f6f4dee06b9cd9ed356936"
server_key_iv: "af96b222e194e5c4ee2fe7858166d59b"
server_key_version: "encryptionkey@0"
I20260620 23:05:06.517647 13631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "4f3c252d77e44e11b14089434aa5099a"
format_stamp: "Formatted at 2026-06-20 23:05:06 on dist-test-slave-7g8g"
server_key: "b6045a0837f6f4dee06b9cd9ed356936"
server_key_iv: "af96b222e194e5c4ee2fe7858166d59b"
server_key_version: "encryptionkey@0"
I20260620 23:05:06.522554 13631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:05:06.525956 13646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:06.527415 13631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:06.527547 13631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "4f3c252d77e44e11b14089434aa5099a"
format_stamp: "Formatted at 2026-06-20 23:05:06 on dist-test-slave-7g8g"
server_key: "b6045a0837f6f4dee06b9cd9ed356936"
server_key_iv: "af96b222e194e5c4ee2fe7858166d59b"
server_key_version: "encryptionkey@0"
I20260620 23:05:06.527726 13631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:06.554868 13631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:06.559427 13631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:06.559638 13631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:06.562101 13631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:06.562182 13631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:06.562289 13631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:06.562340 13631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:05:06 dist-test-slave-7g8g krb5kdc[13375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996706, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:06.584498 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:06.562730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:57309 (local address 127.12.2.126:43961)
0620 23:05:06.562922 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:06.562959 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:06.564390 (+ 1431us) server_negotiation.cc:407] Connection header received
0620 23:05:06.566350 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:06.566355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:06.566483 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:06.566580 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:06.568987 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.569901 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.572684 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.573200 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.573326 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:06.577614 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:06.577619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:06.577623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:06.577656 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:06.580651 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:06.581669 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:06.581674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:06.581678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:06.581828 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:06.582284 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:06.582289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:06.582292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:06.582622 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:06.582764 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:06.583839 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0620 23:05:06.584110 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:05:06.585495 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:06.563291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43961 (local address 127.12.2.66:57309)
0620 23:05:06.564136 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:06.564188 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:06.566087 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:06.566961 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:06.566972 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:06.567808 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:06.568770 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:06.568799 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.570072 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.570081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:06.572529 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:06.572545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.573453 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.573460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:06.573695 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:06.574446 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:06.574480 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:06.577369 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:06.580817 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:06.580825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:06.580847 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:06.581540 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:06.581958 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:06.581962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:06.581964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:06.582170 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:06.582825 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:06.582831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:06.583376 (+ 545us) client_negotiation.cc:769] Sending connection context
0620 23:05:06.583706 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 23:05:06.584155 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":354,"threads_started":1}
I20260620 23:05:06.604836 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:06.591191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:40325 (local address 127.12.2.126:43961)
0620 23:05:06.591338 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:06.591341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:06.591853 (+ 512us) server_negotiation.cc:407] Connection header received
0620 23:05:06.592001 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:06.592007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:06.592092 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:06.592195 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:06.593060 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.594257 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.595795 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.596422 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.596578 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:06.598731 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:06.598739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:06.598745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:06.598784 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:06.602101 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:06.602687 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:06.602692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:06.602696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:06.602845 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:06.603275 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:06.603280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:06.603284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:06.603725 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:06.603884 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:06.604306 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 23:05:06.604521 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:05:06.605047 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:06.591223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43961 (local address 127.12.2.66:40325)
0620 23:05:06.591773 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:06.591787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:06.591893 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:06.592205 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:06.592208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:06.592618 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:06.592936 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:06.592946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.594367 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.594370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:06.595667 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:06.595679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:06.596724 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:06.596732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:06.596900 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:06.597580 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:06.597599 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:06.598553 (+ 954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:06.602221 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:06.602225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:06.602228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:06.602593 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:06.602998 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:06.603007 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:06.603011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:06.603167 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:06.603852 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:06.603855 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:06.604044 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 23:05:06.604201 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 23:05:06.604536 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":319,"threads_started":1}
I20260620 23:05:06.609257 13631 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:32951
I20260620 23:05:06.609308 13765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:32951 every 8 connection(s)
I20260620 23:05:06.610379 13631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:06.612051 13766 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43961
I20260620 23:05:06.612226 13766 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:06.612782 13766 heartbeater.cc:511] Master 127.12.2.126:43961 requested a full tablet report, sending...
I20260620 23:05:06.613981 13420 ts_manager.cc:194] Registered new tserver with Master: 4f3c252d77e44e11b14089434aa5099a (127.12.2.66:32951)
I20260620 23:05:06.614769 13420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:57309
I20260620 23:05:06.615527 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 13631
I20260620 23:05:06.615734 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:06.616255 12297 external_mini_cluster.cc:1428] Setting key 9c2e70221ddcdef4ca41b6f3c71f431c
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:05:06.651764 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43961
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:06.952493 13770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:06.952771 13770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:06.952898 13770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:06.960027 13770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:06.960141 13770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:06.960264 13770 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:05:06.960371 13770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:06.969617 13770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.2.126:43961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:06.970803 13770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:06.972069 13770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:06.983945 13775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:06.984030 13776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:06.983970 13778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:06.985172 13770 server_base.cc:1034] running on GCE node
I20260620 23:05:06.985690 13770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:06.986479 13770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:06.987673 13770 hybrid_clock.cc:648] HybridClock initialized: now 1781996706987613 us; error 40 us; skew 500 ppm
I20260620 23:05:06.987983 13770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:06 dist-test-slave-7g8g krb5kdc[13375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996706, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:06.994089 13770 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:05:06.995637 13770 webserver.cc:460] Webserver started at http://127.12.2.67:45291/ using document root <none> and password file <none>
I20260620 23:05:06.996299 13770 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:06.996404 13770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:06.996709 13770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:06.999498 13770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "5285b930386a4389a7b5b3f8a7c93711"
format_stamp: "Formatted at 2026-06-20 23:05:06 on dist-test-slave-7g8g"
server_key: "69f8a78fc46f8c5726147b7b7ada12bf"
server_key_iv: "7d9de9aac1c11fce37cec0fb10c57d0d"
server_key_version: "encryptionkey@0"
I20260620 23:05:07.000204 13770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "5285b930386a4389a7b5b3f8a7c93711"
format_stamp: "Formatted at 2026-06-20 23:05:06 on dist-test-slave-7g8g"
server_key: "69f8a78fc46f8c5726147b7b7ada12bf"
server_key_iv: "7d9de9aac1c11fce37cec0fb10c57d0d"
server_key_version: "encryptionkey@0"
I20260620 23:05:07.005141 13770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:05:07.008656 13785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:07.009949 13770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:05:07.010097 13770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "5285b930386a4389a7b5b3f8a7c93711"
format_stamp: "Formatted at 2026-06-20 23:05:06 on dist-test-slave-7g8g"
server_key: "69f8a78fc46f8c5726147b7b7ada12bf"
server_key_iv: "7d9de9aac1c11fce37cec0fb10c57d0d"
server_key_version: "encryptionkey@0"
I20260620 23:05:07.010288 13770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:07.021957 13770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:07.025745 13770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:07.025981 13770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:07.028409 13770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:07.028488 13770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:07.028616 13770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:07.028720 13770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 23:05:07 dist-test-slave-7g8g krb5kdc[13375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996706, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:07.048192 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.028818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:60727 (local address 127.12.2.126:43961)
0620 23:05:07.028994 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.029000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.030205 (+ 1205us) server_negotiation.cc:407] Connection header received
0620 23:05:07.032290 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.032296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.032373 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.032454 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:07.034416 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.035616 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.037155 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.037782 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.037909 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.041666 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:07.041674 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:07.041680 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:07.041724 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:07.044932 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.045909 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.045915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.045919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.046057 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.046520 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.046523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.046524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.046768 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:07.046871 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.047763 (+ 892us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.047924 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:05:07.049672 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.029279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43961 (local address 127.12.2.67:60727)
0620 23:05:07.030031 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.030076 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.032037 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.032735 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.032747 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.033412 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:07.034176 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.034197 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.035763 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.035767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.037016 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.037027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.038007 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.038014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.038290 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.039041 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:07.039068 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:07.041464 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:07.045073 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:07.045086 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:07.045103 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:07.045773 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:07.046151 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:07.046156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:07.046160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:07.046415 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:07.046878 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:07.046886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:07.047369 (+ 483us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.047687 (+ 318us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.048205 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":306,"threads_started":1}
I20260620 23:05:07.073124 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.060001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:57165 (local address 127.12.2.126:43961)
0620 23:05:07.060169 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.060175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.061086 (+ 911us) server_negotiation.cc:407] Connection header received
0620 23:05:07.061274 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.061281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.061383 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.061489 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:07.062655 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.063636 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.065402 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.065922 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.066036 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.067951 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:07.067957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:07.067960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:07.067990 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:07.070310 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.070908 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.070913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.070917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.071086 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.071496 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.071503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.071507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.071923 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:07.072091 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.072540 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.072737 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:05:07.073173 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.060094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43961 (local address 127.12.2.67:57165)
0620 23:05:07.060962 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.060981 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.061130 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.061544 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.061550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.062071 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:07.062509 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.062523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.063758 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.063764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.065266 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.065283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.066159 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.066167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.066284 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.066807 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:07.066823 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:07.067770 (+ 947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:07.070422 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:07.070427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:07.070440 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:07.070807 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:07.071189 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:07.071194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:07.071198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:07.071361 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:07.072083 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:07.072091 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:07.072297 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.072425 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.072763 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":532,"threads_started":1}
I20260620 23:05:07.084949 13770 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:37711
I20260620 23:05:07.085011 13904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:37711 every 8 connection(s)
I20260620 23:05:07.086205 13770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:07.088212 13905 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43961
I20260620 23:05:07.088433 13905 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:07.089186 13905 heartbeater.cc:511] Master 127.12.2.126:43961 requested a full tablet report, sending...
I20260620 23:05:07.090452 13420 ts_manager.cc:194] Registered new tserver with Master: 5285b930386a4389a7b5b3f8a7c93711 (127.12.2.67:37711)
I20260620 23:05:07.091275 13420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:60727
I20260620 23:05:07.096561 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 13770
I20260620 23:05:07.096726 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:07.097152 12297 external_mini_cluster.cc:1428] Setting key 43d28da5ee45a67d0c3e515150f03895
I20260620 23:05:07.099761 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:05:07 dist-test-slave-7g8g krb5kdc[13375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260620 23:05:07.136204 13627 heartbeater.cc:503] Master 127.12.2.126:43961 was elected leader, sending a full tablet report...
Jun 20 23:05:07 dist-test-slave-7g8g krb5kdc[13375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:07.142261 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.129542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.12.2.126:43961)
0620 23:05:07.129688 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.129693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.130050 (+ 357us) server_negotiation.cc:407] Connection header received
0620 23:05:07.130189 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.130193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.130269 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.130352 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:07.131751 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.132542 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.133728 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.134119 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.134205 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.137906 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:07.137911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:07.137915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:07.137943 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:07.140253 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.140807 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.140810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.140813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.140940 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.141218 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.141221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.141223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.141472 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:07.141638 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.141859 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.142032 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:05:07.145929 13420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:07.152544 13420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:07.186146 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.164425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:32951 (local address 127.0.0.1:44630)
0620 23:05:07.177695 (+ 13270us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.177713 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.177874 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.178946 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.178954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.178984 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:07.179423 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.179435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.180977 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.180982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.183458 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.183473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.185103 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.185109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.185293 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.185351 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.185484 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.185661 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13122,"mutex_wait_us":44,"spinlock_wait_cycles":11648,"thread_start_us":12991,"threads_started":1}
I20260620 23:05:07.186257 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.165200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.12.2.67:37711)
0620 23:05:07.173692 (+ 8492us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.173703 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.178225 (+ 4522us) server_negotiation.cc:407] Connection header received
0620 23:05:07.178308 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.178314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.178502 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.178680 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:07.180105 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.181546 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.184056 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.184983 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.185104 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.185327 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.185819 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.185974 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8317,"thread_start_us":8140,"threads_started":1}
I20260620 23:05:07.186628 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.164221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:37861 (local address 127.0.0.1:47850)
0620 23:05:07.177289 (+ 13068us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.177342 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.177607 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.180103 (+ 2496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.180107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.180132 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:07.180476 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.180487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.181843 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.181847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.184019 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.184032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.185892 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.185898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.186078 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.186110 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.186190 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.186301 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12761,"thread_start_us":13541,"threads_started":1}
I20260620 23:05:07.187160 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.164939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.12.2.66:32951)
0620 23:05:07.174745 (+ 9806us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.174754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.178100 (+ 3346us) server_negotiation.cc:407] Connection header received
0620 23:05:07.178209 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.178217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.178495 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.178678 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:07.179622 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.180853 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.183620 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.184923 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.185046 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.185253 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.185554 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.186863 (+ 1309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9641,"thread_start_us":9501,"threads_started":1}
I20260620 23:05:07.187474 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.164381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.12.2.65:37861)
0620 23:05:07.179590 (+ 15209us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.179598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.179622 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:05:07.179705 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.179713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.179925 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.180105 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:07.180647 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.181710 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.184173 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.185451 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.185567 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.185739 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.186965 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.187121 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15050,"thread_start_us":14923,"threads_started":1}
I20260620 23:05:07.187011 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.164624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:37711 (local address 127.0.0.1:39668)
0620 23:05:07.178000 (+ 13376us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.178021 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.178119 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.178988 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.178992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.179011 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:07.179894 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.179906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.181694 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.181698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.183902 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.183915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.185479 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.185485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.185634 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.185663 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.186503 (+ 840us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.186631 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13271,"mutex_wait_us":96,"spinlock_wait_cycles":10112,"thread_start_us":11860,"threads_started":1}
I20260620 23:05:07.190672 13557 tablet_service.cc:1467] Processing CreateTablet for tablet 5dac8158b009428bb854a106a2438f33 (DEFAULT_TABLE table=test-table [id=c881a2747d1f44c08b5b3ec9d1cd4f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:07.190743 13696 tablet_service.cc:1467] Processing CreateTablet for tablet 5dac8158b009428bb854a106a2438f33 (DEFAULT_TABLE table=test-table [id=c881a2747d1f44c08b5b3ec9d1cd4f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:07.190737 13835 tablet_service.cc:1467] Processing CreateTablet for tablet 5dac8158b009428bb854a106a2438f33 (DEFAULT_TABLE table=test-table [id=c881a2747d1f44c08b5b3ec9d1cd4f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:07.191916 13557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dac8158b009428bb854a106a2438f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:07.192000 13696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dac8158b009428bb854a106a2438f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:07.192027 13835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dac8158b009428bb854a106a2438f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:07.203037 13924 tablet_bootstrap.cc:492] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: Bootstrap starting.
I20260620 23:05:07.205543 13925 tablet_bootstrap.cc:492] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: Bootstrap starting.
I20260620 23:05:07.206909 13924 tablet_bootstrap.cc:654] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:07.207124 13926 tablet_bootstrap.cc:492] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: Bootstrap starting.
I20260620 23:05:07.208209 13924 log.cc:824] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:07.209159 13925 tablet_bootstrap.cc:654] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:07.210233 13925 log.cc:824] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:07.210256 13926 tablet_bootstrap.cc:654] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:07.211391 13926 log.cc:824] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:07.211489 13924 tablet_bootstrap.cc:492] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: No bootstrap required, opened a new log
I20260620 23:05:07.211763 13924 ts_tablet_manager.cc:1397] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 23:05:07.213979 13925 tablet_bootstrap.cc:492] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: No bootstrap required, opened a new log
I20260620 23:05:07.214109 13926 tablet_bootstrap.cc:492] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: No bootstrap required, opened a new log
I20260620 23:05:07.214224 13925 ts_tablet_manager.cc:1397] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 23:05:07.214371 13926 ts_tablet_manager.cc:1397] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260620 23:05:07.221096 13926 raft_consensus.cc:348] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.221452 13926 raft_consensus.cc:374] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:07.221555 13926 raft_consensus.cc:729] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5285b930386a4389a7b5b3f8a7c93711, State: Initialized, Role: FOLLOWER
I20260620 23:05:07.221668 13924 raft_consensus.cc:348] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.222105 13924 raft_consensus.cc:374] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:07.222242 13924 raft_consensus.cc:729] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37acc82fa373490580f86b01fa0d3907, State: Initialized, Role: FOLLOWER
I20260620 23:05:07.222366 13926 consensus_queue.cc:260] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.222960 13924 consensus_queue.cc:260] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.223590 13905 heartbeater.cc:503] Master 127.12.2.126:43961 was elected leader, sending a full tablet report...
I20260620 23:05:07.224133 13925 raft_consensus.cc:348] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.224589 13925 raft_consensus.cc:374] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:07.224323 13926 ts_tablet_manager.cc:1428] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20260620 23:05:07.224718 13925 raft_consensus.cc:729] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3c252d77e44e11b14089434aa5099a, State: Initialized, Role: FOLLOWER
I20260620 23:05:07.225937 13925 consensus_queue.cc:260] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.226186 13924 ts_tablet_manager.cc:1428] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260620 23:05:07.227048 13766 heartbeater.cc:503] Master 127.12.2.126:43961 was elected leader, sending a full tablet report...
I20260620 23:05:07.227789 13925 ts_tablet_manager.cc:1428] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260620 23:05:07.338125 13906 tablet.cc:2369] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:05:07.362228 13767 tablet.cc:2369] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:05:07.379299 13628 tablet.cc:2369] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:07.394083 13932 raft_consensus.cc:482] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:07.394399 13932 raft_consensus.cc:504] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.396111 13932 leader_election.cc:288] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37acc82fa373490580f86b01fa0d3907 (127.12.2.65:37861), 5285b930386a4389a7b5b3f8a7c93711 (127.12.2.67:37711)
I20260620 23:05:07.404537 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.396751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:57223 (local address 127.12.2.67:37711)
0620 23:05:07.396946 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.396952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.397357 (+ 405us) server_negotiation.cc:407] Connection header received
0620 23:05:07.397565 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.397570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.397648 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.397742 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:07.398416 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.399521 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.402278 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.403520 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.403641 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.403739 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.404174 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.404297 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:05:07.404537 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.396496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:45003 (local address 127.12.2.65:37861)
0620 23:05:07.396698 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.396705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.397468 (+ 763us) server_negotiation.cc:407] Connection header received
0620 23:05:07.397599 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.397603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.397690 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.397773 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:07.398481 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.399513 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.402272 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.403516 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.403641 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.403739 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.404175 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.404297 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:05:07.404747 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.396452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:37861 (local address 127.12.2.66:45003)
0620 23:05:07.397388 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.397403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.397500 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.397782 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.397785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.397816 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:07.398333 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.398342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.399647 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.399653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.402120 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.402134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.403755 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.403760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.403980 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.404054 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.404135 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.404291 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":346,"spinlock_wait_cycles":1408,"thread_start_us":504,"threads_started":1}
I20260620 23:05:07.404769 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.396437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:37711 (local address 127.12.2.66:57223)
0620 23:05:07.397250 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.397267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.397419 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.397776 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.397781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.397824 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:07.398279 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.398296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.399647 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.399652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.402122 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.402134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.403736 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.403746 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.403982 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.404050 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.404139 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.404285 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":160,"spinlock_wait_cycles":6784,"thread_start_us":412,"threads_started":1}
I20260620 23:05:07.405786 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dac8158b009428bb854a106a2438f33" candidate_uuid: "4f3c252d77e44e11b14089434aa5099a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37acc82fa373490580f86b01fa0d3907" is_pre_election: true
I20260620 23:05:07.405848 13860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dac8158b009428bb854a106a2438f33" candidate_uuid: "4f3c252d77e44e11b14089434aa5099a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285b930386a4389a7b5b3f8a7c93711" is_pre_election: true
I20260620 23:05:07.406239 13577 raft_consensus.cc:2454] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f3c252d77e44e11b14089434aa5099a in term 0.
I20260620 23:05:07.406296 13860 raft_consensus.cc:2454] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f3c252d77e44e11b14089434aa5099a in term 0.
I20260620 23:05:07.406880 13649 leader_election.cc:302] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37acc82fa373490580f86b01fa0d3907, 4f3c252d77e44e11b14089434aa5099a; no voters:
I20260620 23:05:07.407238 13932 raft_consensus.cc:2781] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:07.407378 13932 raft_consensus.cc:482] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:07.407477 13932 raft_consensus.cc:3037] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:07.409516 13932 raft_consensus.cc:504] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.410053 13932 leader_election.cc:288] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [CANDIDATE]: Term 1 election: Requested vote from peers 37acc82fa373490580f86b01fa0d3907 (127.12.2.65:37861), 5285b930386a4389a7b5b3f8a7c93711 (127.12.2.67:37711)
I20260620 23:05:07.410436 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dac8158b009428bb854a106a2438f33" candidate_uuid: "4f3c252d77e44e11b14089434aa5099a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37acc82fa373490580f86b01fa0d3907"
I20260620 23:05:07.410490 13860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dac8158b009428bb854a106a2438f33" candidate_uuid: "4f3c252d77e44e11b14089434aa5099a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285b930386a4389a7b5b3f8a7c93711"
I20260620 23:05:07.410640 13860 raft_consensus.cc:3037] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:07.410631 13577 raft_consensus.cc:3037] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:07.413628 13860 raft_consensus.cc:2454] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f3c252d77e44e11b14089434aa5099a in term 1.
I20260620 23:05:07.413628 13577 raft_consensus.cc:2454] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f3c252d77e44e11b14089434aa5099a in term 1.
I20260620 23:05:07.414013 13648 leader_election.cc:302] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 37acc82fa373490580f86b01fa0d3907, 4f3c252d77e44e11b14089434aa5099a, 5285b930386a4389a7b5b3f8a7c93711; no voters:
I20260620 23:05:07.414400 13932 raft_consensus.cc:2781] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:07.414811 13932 raft_consensus.cc:686] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 1 LEADER]: Becoming Leader. State: Replica: 4f3c252d77e44e11b14089434aa5099a, State: Running, Role: LEADER
I20260620 23:05:07.415328 13932 consensus_queue.cc:237] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.419615 13418 catalog_manager.cc:5495] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f3c252d77e44e11b14089434aa5099a (127.12.2.66). New cstate: current_term: 1 leader_uuid: "4f3c252d77e44e11b14089434aa5099a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:07.492784 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.487520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.12.2.66:32951)
0620 23:05:07.487686 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.487691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.487752 (+ 61us) server_negotiation.cc:407] Connection header received
0620 23:05:07.488021 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.488025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.488102 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.488219 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:07.488647 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.489427 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.490701 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.491158 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.491256 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.491740 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:07.492273 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:07.492372 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.492493 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.492573 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:05:07.500993 13577 raft_consensus.cc:1261] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907 [term 1 FOLLOWER]: Refusing update from remote peer 4f3c252d77e44e11b14089434aa5099a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:07.500993 13860 raft_consensus.cc:1261] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711 [term 1 FOLLOWER]: Refusing update from remote peer 4f3c252d77e44e11b14089434aa5099a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:07.501961 13935 consensus_queue.cc:1035] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:07.502384 13932 consensus_queue.cc:1035] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:07.532004 13420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:07.532259 13420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:07.535611 13420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec: Sending DeleteTablet for 3 replicas of tablet 5dac8158b009428bb854a106a2438f33
I20260620 23:05:07.536890 13557 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dac8158b009428bb854a106a2438f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:07 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:47850
I20260620 23:05:07.537025 13835 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dac8158b009428bb854a106a2438f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:07 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:39668
I20260620 23:05:07.537806 13696 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dac8158b009428bb854a106a2438f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:07 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:44630
I20260620 23:05:07.549448 13948 tablet_replica.cc:330] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: stopping tablet replica
I20260620 23:05:07.549980 13420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:07.550729 13420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:07.551433 13947 tablet_replica.cc:330] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: stopping tablet replica
I20260620 23:05:07.551906 13947 raft_consensus.cc:2229] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:07.552419 13947 raft_consensus.cc:2258] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:07.556950 13949 tablet_replica.cc:330] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: stopping tablet replica
I20260620 23:05:07.557297 13947 ts_tablet_manager.cc:1905] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:07.557425 13949 raft_consensus.cc:2229] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:07.558187 13949 raft_consensus.cc:2258] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:07.560436 13557 tablet_service.cc:1467] Processing CreateTablet for tablet 973d0bc141884f3da99eb21c96d03ed1 (DEFAULT_TABLE table=test-table [id=767b604d86d24b92b1caaac78e138655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:07.560936 13557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973d0bc141884f3da99eb21c96d03ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:07.561108 13696 tablet_service.cc:1467] Processing CreateTablet for tablet 973d0bc141884f3da99eb21c96d03ed1 (DEFAULT_TABLE table=test-table [id=767b604d86d24b92b1caaac78e138655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:07.561532 13696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973d0bc141884f3da99eb21c96d03ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:07.564653 13947 ts_tablet_manager.cc:1918] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:07.564814 13947 log.cc:1196] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/5dac8158b009428bb854a106a2438f33
I20260620 23:05:07.565361 13947 ts_tablet_manager.cc:1939] T 5dac8158b009428bb854a106a2438f33 P 37acc82fa373490580f86b01fa0d3907: Deleting consensus metadata
I20260620 23:05:07.567342 13835 tablet_service.cc:1467] Processing CreateTablet for tablet 973d0bc141884f3da99eb21c96d03ed1 (DEFAULT_TABLE table=test-table [id=767b604d86d24b92b1caaac78e138655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:07.567461 13948 raft_consensus.cc:2229] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:07.567795 13835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973d0bc141884f3da99eb21c96d03ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:07.567832 13406 catalog_manager.cc:4841] TS 37acc82fa373490580f86b01fa0d3907 (127.12.2.65:37861): tablet 5dac8158b009428bb854a106a2438f33 (table test-table [id=c881a2747d1f44c08b5b3ec9d1cd4f13]) successfully deleted
I20260620 23:05:07.567940 13948 raft_consensus.cc:2258] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:07.569849 13949 ts_tablet_manager.cc:1905] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:07.570277 13924 tablet_bootstrap.cc:492] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907: Bootstrap starting.
I20260620 23:05:07.571831 13924 tablet_bootstrap.cc:654] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:07.572330 13948 ts_tablet_manager.cc:1905] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:07.573873 13926 tablet_bootstrap.cc:492] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711: Bootstrap starting.
I20260620 23:05:07.575501 13926 tablet_bootstrap.cc:654] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:07.585805 13925 tablet_bootstrap.cc:492] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a: Bootstrap starting.
I20260620 23:05:07.586601 13924 tablet_bootstrap.cc:492] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907: No bootstrap required, opened a new log
I20260620 23:05:07.586745 13924 ts_tablet_manager.cc:1397] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.010s
I20260620 23:05:07.587528 13925 tablet_bootstrap.cc:654] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:07.587621 13949 ts_tablet_manager.cc:1918] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:07.587756 13949 log.cc:1196] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/5dac8158b009428bb854a106a2438f33
I20260620 23:05:07.587630 13924 raft_consensus.cc:348] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.587847 13924 raft_consensus.cc:374] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:07.587922 13924 raft_consensus.cc:729] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37acc82fa373490580f86b01fa0d3907, State: Initialized, Role: FOLLOWER
I20260620 23:05:07.588279 13949 ts_tablet_manager.cc:1939] T 5dac8158b009428bb854a106a2438f33 P 4f3c252d77e44e11b14089434aa5099a: Deleting consensus metadata
I20260620 23:05:07.588212 13924 consensus_queue.cc:260] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.589772 13948 ts_tablet_manager.cc:1918] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:07.589937 13948 log.cc:1196] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/5dac8158b009428bb854a106a2438f33
I20260620 23:05:07.590020 13404 catalog_manager.cc:4841] TS 4f3c252d77e44e11b14089434aa5099a (127.12.2.66:32951): tablet 5dac8158b009428bb854a106a2438f33 (table test-table [id=c881a2747d1f44c08b5b3ec9d1cd4f13]) successfully deleted
I20260620 23:05:07.590440 13948 ts_tablet_manager.cc:1939] T 5dac8158b009428bb854a106a2438f33 P 5285b930386a4389a7b5b3f8a7c93711: Deleting consensus metadata
I20260620 23:05:07.592165 13405 catalog_manager.cc:4841] TS 5285b930386a4389a7b5b3f8a7c93711 (127.12.2.67:37711): tablet 5dac8158b009428bb854a106a2438f33 (table test-table [id=c881a2747d1f44c08b5b3ec9d1cd4f13]) successfully deleted
I20260620 23:05:07.592343 13924 ts_tablet_manager.cc:1428] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260620 23:05:07.595579 13926 tablet_bootstrap.cc:492] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711: No bootstrap required, opened a new log
I20260620 23:05:07.595714 13926 ts_tablet_manager.cc:1397] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.005s
I20260620 23:05:07.595944 13925 tablet_bootstrap.cc:492] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a: No bootstrap required, opened a new log
I20260620 23:05:07.596177 13925 ts_tablet_manager.cc:1397] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260620 23:05:07.596450 13926 raft_consensus.cc:348] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.596643 13926 raft_consensus.cc:374] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:07.596757 13926 raft_consensus.cc:729] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5285b930386a4389a7b5b3f8a7c93711, State: Initialized, Role: FOLLOWER
I20260620 23:05:07.597014 13926 consensus_queue.cc:260] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.597317 13925 raft_consensus.cc:348] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.597898 13925 raft_consensus.cc:374] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:07.598136 13925 raft_consensus.cc:729] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3c252d77e44e11b14089434aa5099a, State: Initialized, Role: FOLLOWER
I20260620 23:05:07.598451 13926 ts_tablet_manager.cc:1428] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 23:05:07.598459 13925 consensus_queue.cc:260] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.599664 13925 ts_tablet_manager.cc:1428] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:07.621742 13930 raft_consensus.cc:482] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:07.621975 13930 raft_consensus.cc:504] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.623677 13930 leader_election.cc:288] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37acc82fa373490580f86b01fa0d3907 (127.12.2.65:37861), 4f3c252d77e44e11b14089434aa5099a (127.12.2.66:32951)
I20260620 23:05:07.633038 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.624645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:55355 (local address 127.12.2.65:37861)
0620 23:05:07.624823 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.624830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.625866 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 23:05:07.626117 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.626123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.626197 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.626255 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:07.627000 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.628112 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.630618 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.631926 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.632044 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.632147 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.632655 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.632787 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:05:07.633054 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.624140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:37861 (local address 127.12.2.67:55355)
0620 23:05:07.625761 (+ 1621us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.625780 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.625971 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.626464 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.626469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.626532 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:07.626792 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.626803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.628308 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.628313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.630439 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.630462 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.632248 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.632255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.632475 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.632510 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.632605 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.632733 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1496,"thread_start_us":6948,"threads_started":1}
I20260620 23:05:07.633816 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973d0bc141884f3da99eb21c96d03ed1" candidate_uuid: "5285b930386a4389a7b5b3f8a7c93711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37acc82fa373490580f86b01fa0d3907" is_pre_election: true
I20260620 23:05:07.634070 13577 raft_consensus.cc:2454] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5285b930386a4389a7b5b3f8a7c93711 in term 0.
I20260620 23:05:07.634539 13788 leader_election.cc:302] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37acc82fa373490580f86b01fa0d3907, 5285b930386a4389a7b5b3f8a7c93711; no voters:
I20260620 23:05:07.635001 13930 raft_consensus.cc:2781] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:07.635120 13930 raft_consensus.cc:482] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:07.635219 13930 raft_consensus.cc:3037] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:07.637584 13930 raft_consensus.cc:504] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.637804 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.625111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:38967 (local address 127.12.2.66:32951)
0620 23:05:07.625284 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.625290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.631468 (+ 6178us) server_negotiation.cc:407] Connection header received
0620 23:05:07.631644 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.631650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.631721 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.631773 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:07.632519 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.633579 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.635426 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.636741 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.636880 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.636978 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.637382 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.637511 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:05:07.638484 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.624580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:32951 (local address 127.12.2.67:38967)
0620 23:05:07.631371 (+ 6791us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.631390 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.631505 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.631943 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.631947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.631970 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:07.632328 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.632344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.633688 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.633694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.635278 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.635293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.636981 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.636987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.637180 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.637211 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.638015 (+ 804us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.638176 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6673,"thread_start_us":935,"threads_started":1}
I20260620 23:05:07.638523 13930 leader_election.cc:288] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [CANDIDATE]: Term 1 election: Requested vote from peers 37acc82fa373490580f86b01fa0d3907 (127.12.2.65:37861), 4f3c252d77e44e11b14089434aa5099a (127.12.2.66:32951)
I20260620 23:05:07.638914 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973d0bc141884f3da99eb21c96d03ed1" candidate_uuid: "5285b930386a4389a7b5b3f8a7c93711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37acc82fa373490580f86b01fa0d3907"
I20260620 23:05:07.639158 13577 raft_consensus.cc:3037] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:07.639508 13715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973d0bc141884f3da99eb21c96d03ed1" candidate_uuid: "5285b930386a4389a7b5b3f8a7c93711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3c252d77e44e11b14089434aa5099a"
I20260620 23:05:07.639468 13716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973d0bc141884f3da99eb21c96d03ed1" candidate_uuid: "5285b930386a4389a7b5b3f8a7c93711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3c252d77e44e11b14089434aa5099a" is_pre_election: true
I20260620 23:05:07.639894 13716 raft_consensus.cc:2454] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5285b930386a4389a7b5b3f8a7c93711 in term 0.
I20260620 23:05:07.642030 13577 raft_consensus.cc:2454] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5285b930386a4389a7b5b3f8a7c93711 in term 1.
I20260620 23:05:07.642378 13788 leader_election.cc:302] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37acc82fa373490580f86b01fa0d3907, 5285b930386a4389a7b5b3f8a7c93711; no voters: 4f3c252d77e44e11b14089434aa5099a
I20260620 23:05:07.642798 13930 raft_consensus.cc:2781] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:07.643210 13930 raft_consensus.cc:686] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 1 LEADER]: Becoming Leader. State: Replica: 5285b930386a4389a7b5b3f8a7c93711, State: Running, Role: LEADER
I20260620 23:05:07.643757 13930 consensus_queue.cc:237] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.646893 13418 catalog_manager.cc:5495] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5285b930386a4389a7b5b3f8a7c93711 (127.12.2.67). New cstate: current_term: 1 leader_uuid: "5285b930386a4389a7b5b3f8a7c93711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } health_report { overall_health: HEALTHY } } }
Jun 20 23:05:07 dist-test-slave-7g8g krb5kdc[13375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996705, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:07.703563 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.689254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43961 (local address 127.0.0.1:52744)
0620 23:05:07.689735 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.689759 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.689904 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.690297 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.690302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.691128 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:07.691354 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.691367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.692404 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.692409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.693903 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.693918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.694581 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.694587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.694754 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.695445 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:07.695478 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:07.698182 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:07.701222 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:07.701234 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:07.701242 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:07.701614 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:07.701929 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:07.701934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:07.701938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:07.702090 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:07.702549 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:07.702556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:07.702714 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.702855 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.703145 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":229,"threads_started":1}
I20260620 23:05:07.703619 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.689316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.12.2.126:43961)
0620 23:05:07.690029 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.690033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.690049 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:05:07.690092 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.690096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.690149 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.690234 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:07.691511 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.692283 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.694046 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.694445 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.694531 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.698405 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:07.698410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:07.698414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:07.698441 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:07.701113 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.701714 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.701720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.701723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.701833 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.702183 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.702187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.702191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.702439 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:07.702567 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.702993 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.703270 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":624,"thread_start_us":524,"threads_started":1}
I20260620 23:05:07.708559 13418 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:52744:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:05:07.709697 13418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:07.716382 13557 tablet_service.cc:1467] Processing CreateTablet for tablet b4e0eea498c04fdd9c798519b64faec5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9623eec015ba4ec7b74c9146caf28b42]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:05:07.716511 13835 tablet_service.cc:1467] Processing CreateTablet for tablet b4e0eea498c04fdd9c798519b64faec5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9623eec015ba4ec7b74c9146caf28b42]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:05:07.716960 13557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e0eea498c04fdd9c798519b64faec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:07.716982 13835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e0eea498c04fdd9c798519b64faec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:07.721686 13926 tablet_bootstrap.cc:492] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711: Bootstrap starting.
I20260620 23:05:07.723074 13926 tablet_bootstrap.cc:654] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:07.723917 13924 tablet_bootstrap.cc:492] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907: Bootstrap starting.
I20260620 23:05:07.724627 13696 tablet_service.cc:1467] Processing CreateTablet for tablet b4e0eea498c04fdd9c798519b64faec5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9623eec015ba4ec7b74c9146caf28b42]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:05:07.725184 13696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e0eea498c04fdd9c798519b64faec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:07.725684 13924 tablet_bootstrap.cc:654] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:07.732152 13924 tablet_bootstrap.cc:492] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907: No bootstrap required, opened a new log
I20260620 23:05:07.732312 13924 ts_tablet_manager.cc:1397] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260620 23:05:07.732625 13926 tablet_bootstrap.cc:492] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711: No bootstrap required, opened a new log
I20260620 23:05:07.732769 13926 ts_tablet_manager.cc:1397] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260620 23:05:07.733036 13924 raft_consensus.cc:348] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.733263 13924 raft_consensus.cc:374] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:07.733340 13924 raft_consensus.cc:729] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37acc82fa373490580f86b01fa0d3907, State: Initialized, Role: FOLLOWER
I20260620 23:05:07.733536 13925 tablet_bootstrap.cc:492] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: Bootstrap starting.
I20260620 23:05:07.733438 13926 raft_consensus.cc:348] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.733640 13926 raft_consensus.cc:374] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:07.733701 13926 raft_consensus.cc:729] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5285b930386a4389a7b5b3f8a7c93711, State: Initialized, Role: FOLLOWER
I20260620 23:05:07.733584 13924 consensus_queue.cc:260] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.733916 13926 consensus_queue.cc:260] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.734107 13924 ts_tablet_manager.cc:1428] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:05:07.734431 13926 ts_tablet_manager.cc:1428] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:05:07.734592 13931 tablet_replica.cc:439] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } } }
I20260620 23:05:07.734925 13931 tablet_replica.cc:442] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:07.735260 13925 tablet_bootstrap.cc:654] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:07.735206 13930 tablet_replica.cc:439] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } } }
I20260620 23:05:07.735459 13930 tablet_replica.cc:442] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:07.739374 13925 tablet_bootstrap.cc:492] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: No bootstrap required, opened a new log
I20260620 23:05:07.739554 13925 ts_tablet_manager.cc:1397] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 23:05:07.740310 13925 raft_consensus.cc:348] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.740577 13925 raft_consensus.cc:374] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:07.740677 13925 raft_consensus.cc:729] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3c252d77e44e11b14089434aa5099a, State: Initialized, Role: FOLLOWER
I20260620 23:05:07.741087 13925 consensus_queue.cc:260] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.741529 13932 tablet_replica.cc:439] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } } }
I20260620 23:05:07.741789 13932 tablet_replica.cc:442] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:07.742753 13925 ts_tablet_manager.cc:1428] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:07.764262 13932 raft_consensus.cc:482] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:07.764452 13932 raft_consensus.cc:504] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.765087 13932 leader_election.cc:288] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37acc82fa373490580f86b01fa0d3907 (127.12.2.65:37861), 5285b930386a4389a7b5b3f8a7c93711 (127.12.2.67:37711)
I20260620 23:05:07.765625 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e0eea498c04fdd9c798519b64faec5" candidate_uuid: "4f3c252d77e44e11b14089434aa5099a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37acc82fa373490580f86b01fa0d3907" is_pre_election: true
I20260620 23:05:07.765647 13860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e0eea498c04fdd9c798519b64faec5" candidate_uuid: "4f3c252d77e44e11b14089434aa5099a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285b930386a4389a7b5b3f8a7c93711" is_pre_election: true
I20260620 23:05:07.765872 13577 raft_consensus.cc:2454] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f3c252d77e44e11b14089434aa5099a in term 0.
I20260620 23:05:07.765931 13860 raft_consensus.cc:2454] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f3c252d77e44e11b14089434aa5099a in term 0.
I20260620 23:05:07.766290 13649 leader_election.cc:302] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37acc82fa373490580f86b01fa0d3907, 4f3c252d77e44e11b14089434aa5099a; no voters:
I20260620 23:05:07.766582 13932 raft_consensus.cc:2781] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:07.766695 13932 raft_consensus.cc:482] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:07.766778 13932 raft_consensus.cc:3037] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:07.769680 13932 raft_consensus.cc:504] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.770290 13932 leader_election.cc:288] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [CANDIDATE]: Term 1 election: Requested vote from peers 37acc82fa373490580f86b01fa0d3907 (127.12.2.65:37861), 5285b930386a4389a7b5b3f8a7c93711 (127.12.2.67:37711)
I20260620 23:05:07.770706 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e0eea498c04fdd9c798519b64faec5" candidate_uuid: "4f3c252d77e44e11b14089434aa5099a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37acc82fa373490580f86b01fa0d3907"
I20260620 23:05:07.770741 13860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e0eea498c04fdd9c798519b64faec5" candidate_uuid: "4f3c252d77e44e11b14089434aa5099a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285b930386a4389a7b5b3f8a7c93711"
I20260620 23:05:07.770957 13577 raft_consensus.cc:3037] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:07.770998 13860 raft_consensus.cc:3037] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:07.773026 13860 raft_consensus.cc:2454] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f3c252d77e44e11b14089434aa5099a in term 1.
I20260620 23:05:07.773372 13648 leader_election.cc:302] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f3c252d77e44e11b14089434aa5099a, 5285b930386a4389a7b5b3f8a7c93711; no voters:
I20260620 23:05:07.773645 13932 raft_consensus.cc:2781] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:07.773782 13932 raft_consensus.cc:686] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [term 1 LEADER]: Becoming Leader. State: Replica: 4f3c252d77e44e11b14089434aa5099a, State: Running, Role: LEADER
I20260620 23:05:07.773732 13577 raft_consensus.cc:2454] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f3c252d77e44e11b14089434aa5099a in term 1.
I20260620 23:05:07.774076 13932 consensus_queue.cc:237] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } }
I20260620 23:05:07.775777 13935 tablet_replica.cc:439] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: TxnStatusTablet state changed. Reason: New leader 4f3c252d77e44e11b14089434aa5099a. Latest consensus state: current_term: 1 leader_uuid: "4f3c252d77e44e11b14089434aa5099a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } } }
I20260620 23:05:07.776098 13935 tablet_replica.cc:442] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:05:07.776898 13968 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:05:07.777182 13968 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:05:07.777110 13418 catalog_manager.cc:5495] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f3c252d77e44e11b14089434aa5099a (127.12.2.66). New cstate: current_term: 1 leader_uuid: "4f3c252d77e44e11b14089434aa5099a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:07.801503 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.795776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.12.2.66:32951)
0620 23:05:07.795945 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.795952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.796198 (+ 246us) server_negotiation.cc:407] Connection header received
0620 23:05:07.796489 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.796495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.796600 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.796677 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:07.797071 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.797817 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.799780 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.800316 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.800409 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.800895 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:07.801007 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:07.801072 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.801219 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.801282 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:05:07.801662 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.795879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:32951 (local address 127.0.0.1:51182)
0620 23:05:07.796106 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.796126 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.796280 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.796723 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.796747 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.796760 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:07.796942 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.796954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.797935 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.797941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.799633 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.799648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.800451 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.800457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.800668 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.800739 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:07.801094 (+ 355us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:07.801112 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.801186 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.801331 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:05:07.805395 13860 raft_consensus.cc:1261] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711 [term 1 FOLLOWER]: Refusing update from remote peer 4f3c252d77e44e11b14089434aa5099a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:07.805411 13577 raft_consensus.cc:1261] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907 [term 1 FOLLOWER]: Refusing update from remote peer 4f3c252d77e44e11b14089434aa5099a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:07.805975 13935 consensus_queue.cc:1035] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:07.806231 13932 consensus_queue.cc:1035] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:07.809485 13930 tablet_replica.cc:439] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711: TxnStatusTablet state changed. Reason: New leader 4f3c252d77e44e11b14089434aa5099a. Latest consensus state: current_term: 1 leader_uuid: "4f3c252d77e44e11b14089434aa5099a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } } }
I20260620 23:05:07.809753 13930 tablet_replica.cc:442] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:07.811517 13932 tablet_replica.cc:439] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f3c252d77e44e11b14089434aa5099a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } } }
I20260620 23:05:07.811774 13932 tablet_replica.cc:442] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:05:07.813864 13938 mvcc.cc:205] Tried to move back new op lower bound from 7299058515164889088 to 7299058515045138432. Current Snapshot: MvccSnapshot[applied={T|T < 7299058515164889088 or (T in {7299058515164889088})}]
I20260620 23:05:07.815899 13940 mvcc.cc:205] Tried to move back new op lower bound from 7299058515164889088 to 7299058515045138432. Current Snapshot: MvccSnapshot[applied={T|T < 7299058515164889088}]
I20260620 23:05:07.816242 13930 tablet_replica.cc:439] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f3c252d77e44e11b14089434aa5099a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } } }
I20260620 23:05:07.816498 13930 tablet_replica.cc:442] T b4e0eea498c04fdd9c798519b64faec5 P 5285b930386a4389a7b5b3f8a7c93711: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:07.823489 13931 tablet_replica.cc:439] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907: TxnStatusTablet state changed. Reason: New leader 4f3c252d77e44e11b14089434aa5099a. Latest consensus state: current_term: 1 leader_uuid: "4f3c252d77e44e11b14089434aa5099a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } } }
I20260620 23:05:07.823784 13931 tablet_replica.cc:442] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:07.823838 13935 tablet_replica.cc:439] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f3c252d77e44e11b14089434aa5099a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } } }
I20260620 23:05:07.824074 13935 tablet_replica.cc:442] T b4e0eea498c04fdd9c798519b64faec5 P 4f3c252d77e44e11b14089434aa5099a: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:05:07.825091 13939 mvcc.cc:205] Tried to move back new op lower bound from 7299058515164889088 to 7299058515045138432. Current Snapshot: MvccSnapshot[applied={T|T < 7299058515164889088}]
I20260620 23:05:07.825132 13931 tablet_replica.cc:439] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f3c252d77e44e11b14089434aa5099a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 } } peers { permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 } } peers { permanent_uuid: "5285b930386a4389a7b5b3f8a7c93711" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 37711 } } }
I20260620 23:05:07.825325 13931 tablet_replica.cc:442] T b4e0eea498c04fdd9c798519b64faec5 P 37acc82fa373490580f86b01fa0d3907: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:07.829442 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.821012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.12.2.67:37711)
0620 23:05:07.821179 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.821185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.821202 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:05:07.821293 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.821299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.821442 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.821583 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:07.822215 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.823283 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.826833 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.827412 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.827531 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.828173 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:07.828872 (+ 699us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:07.828976 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.829114 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.829179 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:05:07.845880 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.839640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:60017 (local address 127.12.2.66:32951)
0620 23:05:07.839835 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.839841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.840439 (+ 598us) server_negotiation.cc:407] Connection header received
0620 23:05:07.840643 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.840647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.840712 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.840788 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:07.841396 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.842182 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.844171 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.844654 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.844754 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.845214 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:07.845362 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:07.845430 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.845579 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.845637 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:05:07.846063 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.839732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:32951 (local address 127.12.2.67:60017)
0620 23:05:07.840333 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.840354 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.840517 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.840821 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.840826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.840839 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:07.841193 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.841221 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.842314 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.842320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.844010 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.844024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.844821 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.844828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.844986 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.845055 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:07.845470 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:07.845482 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.845568 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.845702 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":269,"threads_started":1}
I20260620 23:05:07.855902 13577 raft_consensus.cc:1261] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907 [term 1 FOLLOWER]: Refusing update from remote peer 5285b930386a4389a7b5b3f8a7c93711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:07.855985 13715 raft_consensus.cc:3037] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:07.857995 13955 consensus_queue.cc:1035] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37acc82fa373490580f86b01fa0d3907" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:07.859079 13715 raft_consensus.cc:1261] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a [term 1 FOLLOWER]: Refusing update from remote peer 5285b930386a4389a7b5b3f8a7c93711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:07.859812 13955 consensus_queue.cc:1035] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f3c252d77e44e11b14089434aa5099a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:07.892220 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.885624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:48517 (local address 127.12.2.67:37711)
0620 23:05:07.885793 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.885799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.886557 (+ 758us) server_negotiation.cc:407] Connection header received
0620 23:05:07.886822 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.886828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.886912 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.887032 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:07.887668 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.888479 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.890087 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.890719 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.890848 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.891482 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:07.891619 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:07.891699 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.891879 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.891953 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:05:07.892362 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.885691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:37711 (local address 127.12.2.66:48517)
0620 23:05:07.886459 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:07.886482 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:07.886672 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:07.887066 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:07.887071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:07.887083 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:07.887466 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.887479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.888617 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.888621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.889799 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:07.889813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.891026 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.891033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:07.891221 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.891294 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:07.891748 (+ 454us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:07.891759 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:05:07.891836 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 23:05:07.891985 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":392,"threads_started":1}
I20260620 23:05:07.931577 13420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:07.931846 13420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:07.935411 13420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fde20c95a9974a8bbc2fa7af089f19ec: Sending DeleteTablet for 3 replicas of tablet 973d0bc141884f3da99eb21c96d03ed1
I20260620 23:05:07.936230 13557 tablet_service.cc:1514] Processing DeleteTablet for tablet 973d0bc141884f3da99eb21c96d03ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:07 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:47850
I20260620 23:05:07.936545 13947 tablet_replica.cc:330] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907: stopping tablet replica
I20260620 23:05:07.936545 13835 tablet_service.cc:1514] Processing DeleteTablet for tablet 973d0bc141884f3da99eb21c96d03ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:07 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:39668
I20260620 23:05:07.936760 13947 raft_consensus.cc:2229] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:07.936851 13948 tablet_replica.cc:330] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711: stopping tablet replica
I20260620 23:05:07.936951 13947 raft_consensus.cc:2258] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:07.937050 13948 raft_consensus.cc:2229] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:07.937377 13696 tablet_service.cc:1514] Processing DeleteTablet for tablet 973d0bc141884f3da99eb21c96d03ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:07 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:44630
I20260620 23:05:07.937496 13948 raft_consensus.cc:2258] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:07.937651 13949 tablet_replica.cc:330] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a: stopping tablet replica
I20260620 23:05:07.937822 13949 raft_consensus.cc:2229] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:07.938022 13949 raft_consensus.cc:2258] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:07.938637 13947 ts_tablet_manager.cc:1905] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:07.939327 13949 ts_tablet_manager.cc:1905] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:07.939927 13948 ts_tablet_manager.cc:1905] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:07.946643 13948 ts_tablet_manager.cc:1918] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:07.946792 13948 log.cc:1196] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/973d0bc141884f3da99eb21c96d03ed1
I20260620 23:05:07.947142 13949 ts_tablet_manager.cc:1918] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:07.947312 13949 log.cc:1196] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/973d0bc141884f3da99eb21c96d03ed1
I20260620 23:05:07.947770 13949 ts_tablet_manager.cc:1939] T 973d0bc141884f3da99eb21c96d03ed1 P 4f3c252d77e44e11b14089434aa5099a: Deleting consensus metadata
I20260620 23:05:07.949031 13404 catalog_manager.cc:4841] TS 4f3c252d77e44e11b14089434aa5099a (127.12.2.66:32951): tablet 973d0bc141884f3da99eb21c96d03ed1 (table test-table [id=767b604d86d24b92b1caaac78e138655]) successfully deleted
I20260620 23:05:07.953429 13948 ts_tablet_manager.cc:1939] T 973d0bc141884f3da99eb21c96d03ed1 P 5285b930386a4389a7b5b3f8a7c93711: Deleting consensus metadata
I20260620 23:05:07.954289 13947 ts_tablet_manager.cc:1918] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:07.954550 13947 log.cc:1196] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/973d0bc141884f3da99eb21c96d03ed1
I20260620 23:05:07.955145 13947 ts_tablet_manager.cc:1939] T 973d0bc141884f3da99eb21c96d03ed1 P 37acc82fa373490580f86b01fa0d3907: Deleting consensus metadata
I20260620 23:05:07.956161 13405 catalog_manager.cc:4841] TS 5285b930386a4389a7b5b3f8a7c93711 (127.12.2.67:37711): tablet 973d0bc141884f3da99eb21c96d03ed1 (table test-table [id=767b604d86d24b92b1caaac78e138655]) successfully deleted
I20260620 23:05:07.956269 13406 catalog_manager.cc:4841] TS 37acc82fa373490580f86b01fa0d3907 (127.12.2.65:37861): tablet 973d0bc141884f3da99eb21c96d03ed1 (table test-table [id=767b604d86d24b92b1caaac78e138655]) successfully deleted
Jun 20 23:05:07 dist-test-slave-7g8g krb5kdc[13375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:05:07.975310 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.953985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.12.2.65:37861)
0620 23:05:07.954209 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.954215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.957198 (+ 2983us) server_negotiation.cc:407] Connection header received
0620 23:05:07.957386 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.957391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.957471 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.957570 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:07.959577 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.960676 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.962537 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.963196 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.963319 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.968347 (+ 5028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:07.968367 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:07.968388 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:07.968441 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:07.972270 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.973091 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.973102 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.973106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.973255 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.973666 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.973672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.973675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.974110 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:07.974420 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.974688 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.974894 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260620 23:05:07.976109 13622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51798
I20260620 23:05:07.998436 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:07.979626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.12.2.126:43961)
0620 23:05:07.979822 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:07.979828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:07.983466 (+ 3638us) server_negotiation.cc:407] Connection header received
0620 23:05:07.983661 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:07.983667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:07.983754 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:07.983861 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:07.985385 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.986549 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.988383 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:07.989085 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:07.989206 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:07.992121 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:07.992129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:07.992134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:07.992179 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:07.995577 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.996360 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.996366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.996370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.996527 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:07.996915 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:07.996921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:07.996924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:07.997279 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:07.997523 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:07.997874 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 23:05:07.998114 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 23:05:07.999277 13420 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56246
I20260620 23:05:08.001495 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 13491
I20260620 23:05:08.018714 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:08.003365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.12.2.65:37861)
0620 23:05:08.003554 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:08.003560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:08.003969 (+ 409us) server_negotiation.cc:407] Connection header received
0620 23:05:08.004154 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:08.004161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:08.004245 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:08.004348 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:08.005691 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:08.006801 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:08.008727 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:08.009421 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:08.009555 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:08.012259 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:08.012265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:08.012270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:08.012310 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:08.015749 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:08.016532 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:08.016537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:08.016541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:08.016699 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:08.017112 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:08.017116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:08.017120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:08.017425 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:08.017674 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:08.017956 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 23:05:08.018170 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 23:05:08.019364 13622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51814
W20260620 23:05:08.020016 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 13491: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:05:08.020210 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 13491
I20260620 23:05:08.041896 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 13631
Jun 20 23:05:08 dist-test-slave-7g8g krb5kdc[13375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:05:08.061378 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:08.042665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.12.2.66:32951)
0620 23:05:08.042874 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:08.042880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:08.042961 (+ 81us) server_negotiation.cc:407] Connection header received
0620 23:05:08.043194 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:08.043200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:08.043279 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:08.043387 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:08.044813 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:08.046004 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:08.047803 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:08.048449 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:08.048581 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:08.053780 (+ 5199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:08.053800 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:08.053820 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:08.053882 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:08.057857 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:08.058721 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:08.058732 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:08.058736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:08.058898 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:08.059450 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:08.059455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:08.059458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:08.059959 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:08.060259 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:08.060601 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 23:05:08.060834 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 23:05:08.062142 13761 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51184
W20260620 23:05:08.062817 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 13631: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:05:08.063024 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 13631
I20260620 23:05:08.080538 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 13770
Jun 20 23:05:08 dist-test-slave-7g8g krb5kdc[13375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:05:08.099004 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:08.081241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.12.2.67:37711)
0620 23:05:08.081404 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:08.081408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:08.081508 (+ 100us) server_negotiation.cc:407] Connection header received
0620 23:05:08.081656 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:08.081660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:08.081728 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:08.081844 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:08.083368 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:08.084140 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:08.085957 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:08.086633 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:08.086762 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:08.091922 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:08.091937 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:08.091960 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:08.092010 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:08.096053 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:08.096871 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:08.096880 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:08.096884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:08.097061 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:08.097423 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:08.097428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:08.097431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:08.097866 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:08.098213 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:08.098401 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 23:05:08.098614 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 23:05:08.099702 13900 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48482
W20260620 23:05:08.100427 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 13770: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:05:08.100587 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 13770
I20260620 23:05:08.114559 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 13388
I20260620 23:05:08.115666 13460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:08.228755 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 13388
2026-06-20T23:05:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4241 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:08 dist-test-slave-7g8g krb5kdc[13991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:08 dist-test-slave-7g8g krb5kdc[13991](info): set up 2 sockets
Jun 20 23:05:08 dist-test-slave-7g8g krb5kdc[13991](info): commencing operation
krb5kdc: starting...
W20260620 23:05:09.222699 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:09 dist-test-slave-7g8g krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:05:09.392108 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:41193
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:44215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:41193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:09.693148 14004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:09.693451 14004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:09.693552 14004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:09.700829 14004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:09.700953 14004 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:05:09.701007 14004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:09.701073 14004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:09.701129 14004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:09.701195 14004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:09.710973 14004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:44215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:41193
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:41193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:09.712232 14004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:09.713487 14004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:09.724493 14009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:09.724570 14012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:09.724570 14010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:09.725391 14004 server_base.cc:1034] running on GCE node
I20260620 23:05:09.726050 14004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:09.727352 14004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:09.728551 14004 hybrid_clock.cc:648] HybridClock initialized: now 1781996709728489 us; error 37 us; skew 500 ppm
I20260620 23:05:09.728871 14004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:09 dist-test-slave-7g8g krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996709, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:09.731814 14004 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:05:09.733148 14004 webserver.cc:460] Webserver started at http://127.12.2.126:41723/ using document root <none> and password file <none>
I20260620 23:05:09.733784 14004 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:09.733884 14004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:09.734161 14004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:09.736922 14004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "e576f91dd337497ab82b5d05473a69a4"
format_stamp: "Formatted at 2026-06-20 23:05:09 on dist-test-slave-7g8g"
server_key: "f9c4d53871849acaea4cbb388022687c"
server_key_iv: "131e50374d303d61326cd1e75cc909dd"
server_key_version: "encryptionkey@0"
I20260620 23:05:09.737614 14004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "e576f91dd337497ab82b5d05473a69a4"
format_stamp: "Formatted at 2026-06-20 23:05:09 on dist-test-slave-7g8g"
server_key: "f9c4d53871849acaea4cbb388022687c"
server_key_iv: "131e50374d303d61326cd1e75cc909dd"
server_key_version: "encryptionkey@0"
I20260620 23:05:09.742233 14004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 23:05:09.745440 14019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:09.746615 14004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:09.746773 14004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "e576f91dd337497ab82b5d05473a69a4"
format_stamp: "Formatted at 2026-06-20 23:05:09 on dist-test-slave-7g8g"
server_key: "f9c4d53871849acaea4cbb388022687c"
server_key_iv: "131e50374d303d61326cd1e75cc909dd"
server_key_version: "encryptionkey@0"
I20260620 23:05:09.746974 14004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:09.770951 14004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:09.774657 14004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:09.774896 14004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:09.794140 14004 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:41193
I20260620 23:05:09.794147 14080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:41193 every 8 connection(s)
I20260620 23:05:09.795528 14004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:09.796552 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 14004
I20260620 23:05:09.796730 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:09.797358 12297 external_mini_cluster.cc:1428] Setting key d3eeff125baeb0e0c0669112aa084256
I20260620 23:05:09.799702 14081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:05:09 dist-test-slave-7g8g krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:09.812007 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4: Bootstrap starting.
I20260620 23:05:09.815587 14081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:09.816803 14081 log.cc:824] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:09.819878 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4: No bootstrap required, opened a new log
I20260620 23:05:09.819906 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:09.799269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.12.2.126:41193)
0620 23:05:09.800050 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:09.800065 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:09.800112 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:05:09.802210 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:09.802236 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:09.802740 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:09.803243 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:09.804396 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:09.806360 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:09.807564 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:09.808530 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:09.808707 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:09.812594 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:09.812601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:09.812616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:09.812662 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:09.815807 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:09.816575 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:09.816581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:09.816587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:09.816761 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:09.817154 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:09.817157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:09.817159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:09.817886 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:09.818159 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:09.818541 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 23:05:09.818896 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":332,"threads_started":1}
I20260620 23:05:09.827061 14081 raft_consensus.cc:348] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576f91dd337497ab82b5d05473a69a4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41193 } }
I20260620 23:05:09.827486 14081 raft_consensus.cc:374] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:09.827579 14081 raft_consensus.cc:729] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e576f91dd337497ab82b5d05473a69a4, State: Initialized, Role: FOLLOWER
I20260620 23:05:09.828171 14081 consensus_queue.cc:260] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576f91dd337497ab82b5d05473a69a4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41193 } }
I20260620 23:05:09.828367 14081 raft_consensus.cc:388] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:09.828500 14081 raft_consensus.cc:482] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:09.828642 14081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:09.830673 14081 raft_consensus.cc:504] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576f91dd337497ab82b5d05473a69a4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41193 } }
I20260620 23:05:09.831223 14081 leader_election.cc:302] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e576f91dd337497ab82b5d05473a69a4; no voters:
I20260620 23:05:09.831773 14081 leader_election.cc:288] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:09.831936 14086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:09.833032 14086 raft_consensus.cc:686] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [term 1 LEADER]: Becoming Leader. State: Replica: e576f91dd337497ab82b5d05473a69a4, State: Running, Role: LEADER
I20260620 23:05:09.833631 14086 consensus_queue.cc:237] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576f91dd337497ab82b5d05473a69a4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41193 } }
I20260620 23:05:09.833907 14081 sys_catalog.cc:564] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:09.836367 14087 sys_catalog.cc:455] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e576f91dd337497ab82b5d05473a69a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576f91dd337497ab82b5d05473a69a4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41193 } } }
I20260620 23:05:09.836393 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e576f91dd337497ab82b5d05473a69a4. Latest consensus state: current_term: 1 leader_uuid: "e576f91dd337497ab82b5d05473a69a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576f91dd337497ab82b5d05473a69a4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41193 } } }
I20260620 23:05:09.836833 14087 sys_catalog.cc:458] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:09.836923 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:09.839874 14092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:09.845111 14092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:09.853106 14092 catalog_manager.cc:1347] Generated new cluster ID: c3dcb60747c743229b9bf7cfd1ad555d
I20260620 23:05:09.853215 14092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:09.882146 14092 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:09.883548 14092 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:09.889657 14092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4: Generated new TSK 0
I20260620 23:05:09.890487 14092 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:05:09.938124 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:41193
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:44215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:10.241016 14107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:10.241328 14107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:10.241470 14107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:10.248555 14107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:10.248679 14107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:10.248811 14107 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:05:10.248879 14107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:05:10.248989 14107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:10.258915 14107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:44215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.2.126:41193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:10.260192 14107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:10.261495 14107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:10.273442 14112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:10.273562 14115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:10.273483 14113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:10.274135 14107 server_base.cc:1034] running on GCE node
I20260620 23:05:10.274691 14107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:10.275563 14107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:10.276748 14107 hybrid_clock.cc:648] HybridClock initialized: now 1781996710276698 us; error 46 us; skew 500 ppm
I20260620 23:05:10.277032 14107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:10 dist-test-slave-7g8g krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996710, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:10.279959 14107 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:05:10.281311 14107 webserver.cc:460] Webserver started at http://127.12.2.65:33389/ using document root <none> and password file <none>
I20260620 23:05:10.281916 14107 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:10.282011 14107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:10.282264 14107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:10.284984 14107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "eaa332d5f0e64e03a248d2d2f2691b63"
format_stamp: "Formatted at 2026-06-20 23:05:10 on dist-test-slave-7g8g"
server_key: "ea3e9c951b4831c788244173e89c9e85"
server_key_iv: "de273b946d05d8bf1dcaabfdbed2dd16"
server_key_version: "encryptionkey@0"
I20260620 23:05:10.285650 14107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "eaa332d5f0e64e03a248d2d2f2691b63"
format_stamp: "Formatted at 2026-06-20 23:05:10 on dist-test-slave-7g8g"
server_key: "ea3e9c951b4831c788244173e89c9e85"
server_key_iv: "de273b946d05d8bf1dcaabfdbed2dd16"
server_key_version: "encryptionkey@0"
I20260620 23:05:10.290342 14107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 23:05:10.293521 14122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:10.294766 14107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:10.294893 14107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "eaa332d5f0e64e03a248d2d2f2691b63"
format_stamp: "Formatted at 2026-06-20 23:05:10 on dist-test-slave-7g8g"
server_key: "ea3e9c951b4831c788244173e89c9e85"
server_key_iv: "de273b946d05d8bf1dcaabfdbed2dd16"
server_key_version: "encryptionkey@0"
I20260620 23:05:10.295140 14107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:10.306573 14107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:10.310436 14107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:10.310684 14107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:10.313220 14107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:10.313320 14107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:10.313427 14107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:10.313508 14107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:05:10 dist-test-slave-7g8g krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996710, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:10.334008 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:10.314052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:38623 (local address 127.12.2.126:41193)
0620 23:05:10.314231 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:10.314236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:10.315551 (+ 1315us) server_negotiation.cc:407] Connection header received
0620 23:05:10.317609 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:10.317616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:10.317697 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:10.317784 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:10.319976 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.321043 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.323530 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.324078 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.324191 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:10.328310 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:10.328314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:10.328318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:10.328350 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:10.330688 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:10.331551 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:10.331558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:10.331562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:10.331722 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:10.332154 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:10.332159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:10.332163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:10.332559 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:10.332702 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:10.333377 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 23:05:10.333626 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:05:10.334906 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:10.314596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41193 (local address 127.12.2.65:38623)
0620 23:05:10.315403 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:10.315440 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:10.317345 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:10.318012 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:10.318020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:10.318736 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:10.319644 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:10.319673 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.321209 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.321222 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:10.323328 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:10.323347 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.324344 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.324352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:10.324640 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:10.325316 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:10.325350 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:10.328104 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:10.330813 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:10.330821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:10.330836 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:10.331395 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:10.331856 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:10.331860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:10.331862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:10.332051 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:10.332705 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:10.332711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:10.333050 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 23:05:10.333265 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 23:05:10.333621 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":379,"threads_started":1}
I20260620 23:05:10.354080 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:10.341173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41193 (local address 127.12.2.65:60143)
0620 23:05:10.342041 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:10.342056 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:10.342168 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:10.342440 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:10.342444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:10.342857 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:10.343253 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:10.343264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.344515 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.344520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:10.346019 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:10.346031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.346693 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.346699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:10.346872 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:10.347453 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:10.347469 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:10.348144 (+ 675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:10.351417 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:10.351422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:10.351426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:10.352024 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:10.352403 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:10.352411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:10.352415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:10.352591 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:10.353251 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:10.353254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:10.353415 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 23:05:10.353528 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:05:10.353739 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":419,"threads_started":1}
I20260620 23:05:10.354208 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:10.341699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:60143 (local address 127.12.2.126:41193)
0620 23:05:10.341852 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:10.341857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:10.342127 (+ 270us) server_negotiation.cc:407] Connection header received
0620 23:05:10.342268 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:10.342274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:10.342335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:10.342408 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:10.343428 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.344307 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.346137 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.346528 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.346632 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:10.348313 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:10.348319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:10.348324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:10.348360 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:10.351285 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:10.352146 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:10.352152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:10.352155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:10.352293 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:10.352732 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:10.352737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:10.352741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:10.353130 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:10.353286 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:10.353637 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 23:05:10.353853 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:05:10.363569 14107 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:39907
I20260620 23:05:10.363606 14241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:39907 every 8 connection(s)
I20260620 23:05:10.364698 14107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:10.366485 14242 heartbeater.cc:347] Connected to a master server at 127.12.2.126:41193
I20260620 23:05:10.366706 14242 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:10.367388 14242 heartbeater.cc:511] Master 127.12.2.126:41193 requested a full tablet report, sending...
I20260620 23:05:10.369161 14036 ts_manager.cc:194] Registered new tserver with Master: eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65:39907)
I20260620 23:05:10.370541 14036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:38623
I20260620 23:05:10.373461 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 14107
I20260620 23:05:10.373643 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:10.374213 12297 external_mini_cluster.cc:1428] Setting key c014b6bf31621beda20e6b59c2b6b4af
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:05:10.407505 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:41193
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:44215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:10.711345 14246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:10.711606 14246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:10.711711 14246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:10.718634 14246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:10.718739 14246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:10.718820 14246 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:05:10.718870 14246 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:05:10.718955 14246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:10.728456 14246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:44215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.2.126:41193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:10.729617 14246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:10.730867 14246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:10.743180 14254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:10.743216 14252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:10.743254 14251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:10.744007 14246 server_base.cc:1034] running on GCE node
I20260620 23:05:10.744539 14246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:10.745339 14246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:10.746551 14246 hybrid_clock.cc:648] HybridClock initialized: now 1781996710746494 us; error 42 us; skew 500 ppm
I20260620 23:05:10.746845 14246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:10 dist-test-slave-7g8g krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996710, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:10.749652 14246 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:05:10.751085 14246 webserver.cc:460] Webserver started at http://127.12.2.66:36555/ using document root <none> and password file <none>
I20260620 23:05:10.751719 14246 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:10.751827 14246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:10.752097 14246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:10.754772 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5"
format_stamp: "Formatted at 2026-06-20 23:05:10 on dist-test-slave-7g8g"
server_key: "ce7e0ef968a14f2351ed7f1367bcde02"
server_key_iv: "73633a6c8ffb95ee1ead2b08449d5d4a"
server_key_version: "encryptionkey@0"
I20260620 23:05:10.755504 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5"
format_stamp: "Formatted at 2026-06-20 23:05:10 on dist-test-slave-7g8g"
server_key: "ce7e0ef968a14f2351ed7f1367bcde02"
server_key_iv: "73633a6c8ffb95ee1ead2b08449d5d4a"
server_key_version: "encryptionkey@0"
I20260620 23:05:10.760183 14246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:05:10.763464 14261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:10.764739 14246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:05:10.764871 14246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5"
format_stamp: "Formatted at 2026-06-20 23:05:10 on dist-test-slave-7g8g"
server_key: "ce7e0ef968a14f2351ed7f1367bcde02"
server_key_iv: "73633a6c8ffb95ee1ead2b08449d5d4a"
server_key_version: "encryptionkey@0"
I20260620 23:05:10.765038 14246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:10.775070 14246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:10.778498 14246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:10.778784 14246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:10.781059 14246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:10.781153 14246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:10.781253 14246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:10.781322 14246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:05:10 dist-test-slave-7g8g krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996710, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:10.803107 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:10.781671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:36073 (local address 127.12.2.126:41193)
0620 23:05:10.781852 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:10.781859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:10.783114 (+ 1255us) server_negotiation.cc:407] Connection header received
0620 23:05:10.785005 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:10.785013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:10.785117 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:10.785222 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:10.787152 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.788131 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.790839 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.791420 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.791551 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:10.796116 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:10.796124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:10.796129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:10.796174 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:10.799729 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:10.800534 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:10.800539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:10.800543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:10.800704 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:10.801169 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:10.801174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:10.801178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:10.801535 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:10.801675 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:10.802410 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 23:05:10.802661 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:05:10.804158 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:10.782144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41193 (local address 127.12.2.66:36073)
0620 23:05:10.782915 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:10.782989 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:10.784754 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:10.785462 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:10.785473 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:10.786113 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:10.786892 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:10.786920 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.788289 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.788296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:10.790686 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:10.790702 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.791664 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.791671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:10.791981 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:10.792772 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:10.792807 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:10.795861 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:10.799859 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:10.799868 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:10.799884 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:10.800419 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:10.800810 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:10.800815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:10.800818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:10.801060 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:10.801650 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:10.801658 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:10.802047 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 23:05:10.802299 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 23:05:10.802776 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":344,"threads_started":1}
I20260620 23:05:10.832770 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:10.817981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:47329 (local address 127.12.2.126:41193)
0620 23:05:10.818169 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:10.818175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:10.819231 (+ 1056us) server_negotiation.cc:407] Connection header received
0620 23:05:10.819466 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:10.819472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:10.819553 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:10.819668 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:10.820936 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.821922 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.823888 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.824586 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.824720 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:10.826415 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:10.826422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:10.826428 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:10.826469 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:10.829749 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:10.830568 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:10.830573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:10.830577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:10.830693 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:10.831180 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:10.831185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:10.831188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:10.831601 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:10.831798 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:10.832224 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 23:05:10.832420 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:05:10.832865 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:10.818048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41193 (local address 127.12.2.66:47329)
0620 23:05:10.819139 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:10.819158 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:10.819314 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:10.819697 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:10.819702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:10.820341 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:10.820796 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:10.820810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.822085 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.822091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:10.823738 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:10.823756 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:10.824759 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:10.824763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:10.824875 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:10.825542 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:10.825560 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:10.826240 (+ 680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:10.829875 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:10.829879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:10.829883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:10.830452 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:10.830803 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:10.830809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:10.830813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:10.831016 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:10.831761 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:10.831766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:10.831980 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 23:05:10.832107 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:05:10.832414 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":758,"threads_started":1}
I20260620 23:05:10.837321 14246 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:38591
I20260620 23:05:10.837349 14380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:38591 every 8 connection(s)
I20260620 23:05:10.838465 14246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:10.840261 14381 heartbeater.cc:347] Connected to a master server at 127.12.2.126:41193
I20260620 23:05:10.840461 14381 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:10.841068 14381 heartbeater.cc:511] Master 127.12.2.126:41193 requested a full tablet report, sending...
I20260620 23:05:10.842260 14036 ts_manager.cc:194] Registered new tserver with Master: ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591)
I20260620 23:05:10.842772 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 14246
I20260620 23:05:10.842916 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:10.843156 14036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:36073
I20260620 23:05:10.843398 12297 external_mini_cluster.cc:1428] Setting key e45424d3428b65097bc755394d96f428
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:05:10.877882 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:41193
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:44215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:11.177637 14385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:11.177884 14385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:11.177989 14385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:11.184834 14385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:11.184927 14385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:11.185012 14385 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:05:11.185062 14385 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:05:11.185127 14385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:11.194391 14385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:44215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.2.126:41193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:11.195577 14385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:11.196789 14385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:11.208904 14390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:11.208964 14391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:11.209160 14393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:11.209970 14385 server_base.cc:1034] running on GCE node
I20260620 23:05:11.210521 14385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:11.211439 14385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:11.212641 14385 hybrid_clock.cc:648] HybridClock initialized: now 1781996711212591 us; error 44 us; skew 500 ppm
I20260620 23:05:11.212930 14385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:11 dist-test-slave-7g8g krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996711, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:11.215716 14385 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:05:11.217048 14385 webserver.cc:460] Webserver started at http://127.12.2.67:39733/ using document root <none> and password file <none>
I20260620 23:05:11.217648 14385 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:11.217757 14385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:11.218040 14385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:11.220764 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "0860c2283ba84e0f80fcfa0a4efc7201"
format_stamp: "Formatted at 2026-06-20 23:05:11 on dist-test-slave-7g8g"
server_key: "20b5e2b753033c43fcfc5e4376aec2c3"
server_key_iv: "011db3e88c0cab3ea595a64df4cc9517"
server_key_version: "encryptionkey@0"
I20260620 23:05:11.221437 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "0860c2283ba84e0f80fcfa0a4efc7201"
format_stamp: "Formatted at 2026-06-20 23:05:11 on dist-test-slave-7g8g"
server_key: "20b5e2b753033c43fcfc5e4376aec2c3"
server_key_iv: "011db3e88c0cab3ea595a64df4cc9517"
server_key_version: "encryptionkey@0"
I20260620 23:05:11.226058 14385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:05:11.229220 14400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:11.230437 14385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:11.230583 14385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "0860c2283ba84e0f80fcfa0a4efc7201"
format_stamp: "Formatted at 2026-06-20 23:05:11 on dist-test-slave-7g8g"
server_key: "20b5e2b753033c43fcfc5e4376aec2c3"
server_key_iv: "011db3e88c0cab3ea595a64df4cc9517"
server_key_version: "encryptionkey@0"
I20260620 23:05:11.230748 14385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:11.247607 14385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:11.251294 14385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:11.251538 14385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:11.253912 14385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:11.254014 14385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:11.254123 14385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:11.254191 14385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:05:11 dist-test-slave-7g8g krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996711, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:11.275877 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.254499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:47189 (local address 127.12.2.126:41193)
0620 23:05:11.254684 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.254689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.256033 (+ 1344us) server_negotiation.cc:407] Connection header received
0620 23:05:11.258389 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.258396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.258497 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.258608 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:11.260562 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.261488 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.263856 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.264420 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.264541 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.268754 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:11.268760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:11.268766 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:11.268800 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:11.271767 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:11.272565 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:11.272571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:11.272575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:11.272699 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:11.273200 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:11.273205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:11.273209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:11.273527 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:11.273659 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.275354 (+ 1695us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.275571 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:05:11.277529 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.255053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41193 (local address 127.12.2.67:47189)
0620 23:05:11.255858 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:11.255903 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:11.258029 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:11.258883 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:11.258897 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:11.259566 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:11.260381 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.260410 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.261676 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.261681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.263693 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.263707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.264823 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.264831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.265075 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.265742 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:11.265771 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:11.268511 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:11.271893 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:11.271901 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:11.271919 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:11.272459 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:11.272833 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:11.272838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:11.272841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:11.273088 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:11.273729 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:11.273734 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:11.274162 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 23:05:11.274408 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 23:05:11.274910 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":346,"threads_started":1}
I20260620 23:05:11.289510 14385 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:36543
I20260620 23:05:11.289583 14519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:36543 every 8 connection(s)
I20260620 23:05:11.290786 14385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:11.293370 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 14385
I20260620 23:05:11.293529 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:11.293933 12297 external_mini_cluster.cc:1428] Setting key 0a9fc89d79291669d6d674695c84e8e9
I20260620 23:05:11.293254 14520 heartbeater.cc:347] Connected to a master server at 127.12.2.126:41193
I20260620 23:05:11.294615 14520 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:11.295466 14520 heartbeater.cc:511] Master 127.12.2.126:41193 requested a full tablet report, sending...
I20260620 23:05:11.296149 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.283441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41193 (local address 127.12.2.67:36567)
0620 23:05:11.284036 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:11.284050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:11.284171 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:11.284489 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:11.284494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:11.285168 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:11.285660 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.285679 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.286819 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.286828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.288163 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.288174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.288954 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.288960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.289123 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.289656 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:11.289677 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:11.290362 (+ 685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:11.293532 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:11.293537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:11.293541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:11.293908 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:11.294263 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:11.294268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:11.294272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:11.294406 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:11.295047 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:11.295052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:11.295243 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 23:05:11.295407 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 23:05:11.295715 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":335,"threads_started":1}
I20260620 23:05:11.296963 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.283329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:36567 (local address 127.12.2.126:41193)
0620 23:05:11.283468 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.283472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.284124 (+ 652us) server_negotiation.cc:407] Connection header received
0620 23:05:11.284276 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.284282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.284339 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.284434 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:11.285825 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.286663 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.288291 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.288775 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.288891 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.290525 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:11.290530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:11.290534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:11.290561 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:11.293382 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:11.294017 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:11.294022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:11.294025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:11.294166 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:11.294519 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:11.294524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:11.294527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:11.294848 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:11.295073 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.296447 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.296648 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:05:11.299749 14036 ts_manager.cc:194] Registered new tserver with Master: 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543)
I20260620 23:05:11.300778 14036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:47189
I20260620 23:05:11.308801 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:05:11 dist-test-slave-7g8g krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:05:11 dist-test-slave-7g8g krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:11.351529 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.338052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.12.2.126:41193)
0620 23:05:11.338264 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.338270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.338457 (+ 187us) server_negotiation.cc:407] Connection header received
0620 23:05:11.338668 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.338672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.338739 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.338832 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:11.340368 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.341204 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.342462 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.342896 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.343076 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.346704 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:11.346709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:11.346712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:11.346744 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:11.349004 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:11.349571 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:11.349577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:11.349580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:11.349741 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:11.350105 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:11.350109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:11.350111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:11.350423 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:11.350661 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.350980 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.351199 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:05:11.355201 14036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:11.361671 14036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:11.373413 14242 heartbeater.cc:503] Master 127.12.2.126:41193 was elected leader, sending a full tablet report...
I20260620 23:05:11.396330 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.372953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39907 (local address 127.0.0.1:52326)
0620 23:05:11.384698 (+ 11745us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:11.384740 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:11.384911 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:11.389085 (+ 4174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:11.389092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:11.389153 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:11.389495 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.389508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.391294 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.391299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.393651 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.393665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.395334 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.395340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.395532 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.395583 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:05:11.395742 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 23:05:11.395925 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11481,"spinlock_wait_cycles":10368,"thread_start_us":12150,"threads_started":1}
I20260620 23:05:11.396955 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.373146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:38591 (local address 127.0.0.1:40486)
0620 23:05:11.388990 (+ 15844us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:11.389008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:11.389150 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:11.390101 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:11.390105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:11.390129 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:11.390495 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.390506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.392212 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.392217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.394288 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.394301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.396181 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.396187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.396378 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.396409 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:05:11.396491 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 23:05:11.396604 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15727,"spinlock_wait_cycles":2432,"thread_start_us":16088,"threads_started":1}
I20260620 23:05:11.397195 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.373131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.12.2.65:39907)
0620 23:05:11.388511 (+ 15380us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.388519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.388556 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:05:11.388642 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.388649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.388889 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.389055 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:11.389671 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.391139 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.393865 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.395133 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.395254 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.395456 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.396784 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.396945 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15230,"thread_start_us":16077,"threads_started":1}
I20260620 23:05:11.397590 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.373571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.12.2.66:38591)
0620 23:05:11.388665 (+ 15094us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.388671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.389414 (+ 743us) server_negotiation.cc:407] Connection header received
0620 23:05:11.389519 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.389526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.389748 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.389925 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:11.390672 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.392042 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.394473 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.395764 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.395865 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.396048 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.397227 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.397390 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14955,"thread_start_us":15095,"threads_started":1}
I20260620 23:05:11.400517 14176 tablet_service.cc:1467] Processing CreateTablet for tablet c7e0789de61a4d12b6ea2324411c2493 (DEFAULT_TABLE table=test-table [id=9057d69cc36a4104a83248fe20dea509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:11.400544 14313 tablet_service.cc:1467] Processing CreateTablet for tablet c7e0789de61a4d12b6ea2324411c2493 (DEFAULT_TABLE table=test-table [id=9057d69cc36a4104a83248fe20dea509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:11.401880 14313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e0789de61a4d12b6ea2324411c2493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:11.402350 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.389506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:36543 (local address 127.0.0.1:36014)
0620 23:05:11.394164 (+ 4658us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:11.394180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:11.394291 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:11.394825 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:11.394830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:11.394855 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:11.395207 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.395218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.397145 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.397150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.399832 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.399846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.401584 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.401590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.401749 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.401782 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:05:11.401861 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:05:11.402027 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4550,"thread_start_us":4413,"threads_started":1}
I20260620 23:05:11.402387 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.386241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.12.2.67:36543)
0620 23:05:11.393909 (+ 7668us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.393918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.394282 (+ 364us) server_negotiation.cc:407] Connection header received
0620 23:05:11.394418 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.394426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.394644 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.394821 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:11.395439 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.396988 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.399976 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.401370 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.401501 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.401728 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.401936 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.402092 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7509,"thread_start_us":7349,"threads_started":1}
I20260620 23:05:11.402812 14176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e0789de61a4d12b6ea2324411c2493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:11.406167 14450 tablet_service.cc:1467] Processing CreateTablet for tablet c7e0789de61a4d12b6ea2324411c2493 (DEFAULT_TABLE table=test-table [id=9057d69cc36a4104a83248fe20dea509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:11.407543 14450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e0789de61a4d12b6ea2324411c2493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:11.412743 14539 tablet_bootstrap.cc:492] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: Bootstrap starting.
I20260620 23:05:11.415575 14539 tablet_bootstrap.cc:654] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:11.416477 14539 log.cc:824] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:11.417425 14540 tablet_bootstrap.cc:492] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: Bootstrap starting.
I20260620 23:05:11.420400 14542 tablet_bootstrap.cc:492] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: Bootstrap starting.
I20260620 23:05:11.421139 14540 tablet_bootstrap.cc:654] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:11.422197 14540 log.cc:824] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:11.423362 14542 tablet_bootstrap.cc:654] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:11.423471 14539 tablet_bootstrap.cc:492] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: No bootstrap required, opened a new log
I20260620 23:05:11.423746 14539 ts_tablet_manager.cc:1397] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260620 23:05:11.424310 14542 log.cc:824] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:11.424988 14540 tablet_bootstrap.cc:492] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: No bootstrap required, opened a new log
I20260620 23:05:11.425237 14540 ts_tablet_manager.cc:1397] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 23:05:11.426431 14542 tablet_bootstrap.cc:492] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: No bootstrap required, opened a new log
I20260620 23:05:11.426649 14542 ts_tablet_manager.cc:1397] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 23:05:11.433470 14542 raft_consensus.cc:348] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.433947 14542 raft_consensus.cc:374] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:11.433768 14539 raft_consensus.cc:348] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.434046 14542 raft_consensus.cc:729] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0860c2283ba84e0f80fcfa0a4efc7201, State: Initialized, Role: FOLLOWER
I20260620 23:05:11.434170 14539 raft_consensus.cc:374] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:11.434267 14539 raft_consensus.cc:729] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed74ae9cd6d4465c8a0166bf7bb459e5, State: Initialized, Role: FOLLOWER
I20260620 23:05:11.434705 14542 consensus_queue.cc:260] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.434863 14539 consensus_queue.cc:260] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.435437 14540 raft_consensus.cc:348] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.435844 14520 heartbeater.cc:503] Master 127.12.2.126:41193 was elected leader, sending a full tablet report...
I20260620 23:05:11.436038 14540 raft_consensus.cc:374] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:11.436123 14540 raft_consensus.cc:729] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa332d5f0e64e03a248d2d2f2691b63, State: Initialized, Role: FOLLOWER
I20260620 23:05:11.436558 14542 ts_tablet_manager.cc:1428] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 23:05:11.436878 14540 consensus_queue.cc:260] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.437506 14381 heartbeater.cc:503] Master 127.12.2.126:41193 was elected leader, sending a full tablet report...
I20260620 23:05:11.438117 14539 ts_tablet_manager.cc:1428] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 23:05:11.440585 14540 ts_tablet_manager.cc:1428] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260620 23:05:11.542604 14521 tablet.cc:2369] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:05:11.590277 14382 tablet.cc:2369] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:05:11.616886 14243 tablet.cc:2369] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:11.655655 14546 raft_consensus.cc:482] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:11.655936 14546 raft_consensus.cc:504] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.657521 14546 leader_election.cc:288] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65:39907), 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543)
I20260620 23:05:11.664901 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.657965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:38483 (local address 127.12.2.65:39907)
0620 23:05:11.658139 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.658143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.658567 (+ 424us) server_negotiation.cc:407] Connection header received
0620 23:05:11.658744 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.658751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.658851 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.659005 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:11.659592 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.660529 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.662814 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.663967 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.664106 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.664207 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.664586 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.664682 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:05:11.665089 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.657815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39907 (local address 127.12.2.66:38483)
0620 23:05:11.658422 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:11.658445 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:11.658587 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:11.659047 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:11.659051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:11.659078 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:11.659458 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.659470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.660646 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.660649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.662657 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.662675 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.664164 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.664168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.664280 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.664305 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 23:05:11.664528 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 23:05:11.664665 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":1,"thread_start_us":287,"threads_started":1}
I20260620 23:05:11.665233 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.658569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:36543 (local address 127.12.2.66:58381)
0620 23:05:11.659205 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:11.659224 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:11.659339 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:11.659666 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:11.659669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:11.659688 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:11.659950 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.659956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.661092 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.661098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.662986 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.662997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.664482 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.664489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.664656 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.664684 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:05:11.664765 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:05:11.664879 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":27,"thread_start_us":261,"threads_started":1}
I20260620 23:05:11.665347 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.658748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:58381 (local address 127.12.2.67:36543)
0620 23:05:11.659042 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.659048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.659403 (+ 355us) server_negotiation.cc:407] Connection header received
0620 23:05:11.659478 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.659481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.659551 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.659657 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:11.660076 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.660934 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.663112 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.664259 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.664373 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.664468 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.664800 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.664915 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:05:11.666096 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e0789de61a4d12b6ea2324411c2493" candidate_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" is_pre_election: true
I20260620 23:05:11.666127 14197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e0789de61a4d12b6ea2324411c2493" candidate_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" is_pre_election: true
I20260620 23:05:11.666545 14197 raft_consensus.cc:2454] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed74ae9cd6d4465c8a0166bf7bb459e5 in term 0.
I20260620 23:05:11.666548 14470 raft_consensus.cc:2454] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed74ae9cd6d4465c8a0166bf7bb459e5 in term 0.
I20260620 23:05:11.667132 14262 leader_election.cc:302] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaa332d5f0e64e03a248d2d2f2691b63, ed74ae9cd6d4465c8a0166bf7bb459e5; no voters:
I20260620 23:05:11.667461 14546 raft_consensus.cc:2781] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:11.667637 14546 raft_consensus.cc:482] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:11.667739 14546 raft_consensus.cc:3037] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:11.669987 14546 raft_consensus.cc:504] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.670897 14197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e0789de61a4d12b6ea2324411c2493" candidate_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa332d5f0e64e03a248d2d2f2691b63"
I20260620 23:05:11.671115 14197 raft_consensus.cc:3037] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:11.671180 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e0789de61a4d12b6ea2324411c2493" candidate_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0860c2283ba84e0f80fcfa0a4efc7201"
I20260620 23:05:11.671376 14470 raft_consensus.cc:3037] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:11.673202 14197 raft_consensus.cc:2454] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed74ae9cd6d4465c8a0166bf7bb459e5 in term 1.
I20260620 23:05:11.673401 14546 leader_election.cc:288] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [CANDIDATE]: Term 1 election: Requested vote from peers eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65:39907), 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543)
I20260620 23:05:11.673473 14470 raft_consensus.cc:2454] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed74ae9cd6d4465c8a0166bf7bb459e5 in term 1.
I20260620 23:05:11.673532 14262 leader_election.cc:302] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaa332d5f0e64e03a248d2d2f2691b63, ed74ae9cd6d4465c8a0166bf7bb459e5; no voters:
I20260620 23:05:11.673795 14546 raft_consensus.cc:2781] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:11.674193 14546 raft_consensus.cc:686] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 LEADER]: Becoming Leader. State: Replica: ed74ae9cd6d4465c8a0166bf7bb459e5, State: Running, Role: LEADER
I20260620 23:05:11.674614 14546 consensus_queue.cc:237] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.679749 14035 catalog_manager.cc:5495] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66). New cstate: current_term: 1 leader_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:05:11 dist-test-slave-7g8g krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996709, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:11.793666 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.781256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.12.2.126:41193)
0620 23:05:11.781406 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.781410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.781982 (+ 572us) server_negotiation.cc:407] Connection header received
0620 23:05:11.782147 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.782152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.782208 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.782284 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:11.783465 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.784185 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.785381 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.785795 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.785899 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.788855 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:11.788859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:11.788862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:11.788890 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:11.791220 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:11.791837 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:11.791842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:11.791845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:11.791959 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:11.792272 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:11.792277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:11.792280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:11.792517 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:11.792636 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.793015 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.793289 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":45}
I20260620 23:05:11.793704 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.781320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41193 (local address 127.0.0.1:43146)
0620 23:05:11.781883 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:11.781897 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:11.782044 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:11.782319 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:11.782323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:11.783022 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:11.783346 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.783357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.784298 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.784312 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.785261 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.785272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.785931 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.785936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.786110 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.786658 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:11.786674 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:11.788706 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:11.791331 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:11.791343 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:11.791350 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:11.791739 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:11.792051 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:11.792055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:11.792059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:11.792183 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:11.792628 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:11.792636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:11.792782 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 23:05:11.792915 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 23:05:11.793157 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":293,"threads_started":1}
I20260620 23:05:11.798631 14035 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:43146:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:05:11.799741 14035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:11.806111 14176 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0d52a3c10d43bbb870a2d144b7f687 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf011236d1604645852205b8135b51cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:05:11.806306 14313 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0d52a3c10d43bbb870a2d144b7f687 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf011236d1604645852205b8135b51cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:05:11.807101 14313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0d52a3c10d43bbb870a2d144b7f687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:11.807108 14176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0d52a3c10d43bbb870a2d144b7f687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:11.810462 14450 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0d52a3c10d43bbb870a2d144b7f687 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf011236d1604645852205b8135b51cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:05:11.811337 14450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0d52a3c10d43bbb870a2d144b7f687. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:11.812454 14539 tablet_bootstrap.cc:492] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5: Bootstrap starting.
I20260620 23:05:11.813455 14540 tablet_bootstrap.cc:492] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63: Bootstrap starting.
I20260620 23:05:11.814073 14539 tablet_bootstrap.cc:654] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:11.814589 14540 tablet_bootstrap.cc:654] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:11.816694 14540 tablet_bootstrap.cc:492] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63: No bootstrap required, opened a new log
I20260620 23:05:11.816862 14540 ts_tablet_manager.cc:1397] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:11.817467 14540 raft_consensus.cc:348] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.817651 14540 raft_consensus.cc:374] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:11.817703 14539 tablet_bootstrap.cc:492] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5: No bootstrap required, opened a new log
I20260620 23:05:11.817752 14540 raft_consensus.cc:729] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa332d5f0e64e03a248d2d2f2691b63, State: Initialized, Role: FOLLOWER
I20260620 23:05:11.817893 14539 ts_tablet_manager.cc:1397] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 23:05:11.817968 14542 tablet_bootstrap.cc:492] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: Bootstrap starting.
I20260620 23:05:11.818056 14540 consensus_queue.cc:260] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.818604 14540 ts_tablet_manager.cc:1428] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:11.818651 14539 raft_consensus.cc:348] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.818513 14547 tablet_replica.cc:439] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } } }
I20260620 23:05:11.818884 14539 raft_consensus.cc:374] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:11.818924 14547 tablet_replica.cc:442] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:11.818989 14539 raft_consensus.cc:729] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed74ae9cd6d4465c8a0166bf7bb459e5, State: Initialized, Role: FOLLOWER
I20260620 23:05:11.819216 14539 consensus_queue.cc:260] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.819499 14542 tablet_bootstrap.cc:654] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:11.819630 14546 tablet_replica.cc:439] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } } }
I20260620 23:05:11.819880 14546 tablet_replica.cc:442] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:11.820669 14539 ts_tablet_manager.cc:1428] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 23:05:11.821797 14542 tablet_bootstrap.cc:492] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: No bootstrap required, opened a new log
I20260620 23:05:11.821913 14542 ts_tablet_manager.cc:1397] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:05:11.822428 14542 raft_consensus.cc:348] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.822609 14542 raft_consensus.cc:374] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:11.822695 14542 raft_consensus.cc:729] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0860c2283ba84e0f80fcfa0a4efc7201, State: Initialized, Role: FOLLOWER
I20260620 23:05:11.822886 14542 consensus_queue.cc:260] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.823431 14542 ts_tablet_manager.cc:1428] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 23:05:11.823323 14545 tablet_replica.cc:439] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } } }
I20260620 23:05:11.823637 14545 tablet_replica.cc:442] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:11.912663 14545 raft_consensus.cc:482] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:11.912873 14545 raft_consensus.cc:504] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.914315 14545 leader_election.cc:288] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65:39907), ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591)
I20260620 23:05:11.922520 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.914770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:42703 (local address 127.12.2.66:38591)
0620 23:05:11.915776 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.915781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.915796 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:05:11.915854 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.915858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.915920 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.916008 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:11.916637 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.917705 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.920282 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.921450 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.921602 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.921703 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.922093 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.922228 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":781,"threads_started":1}
I20260620 23:05:11.922506 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.914734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:42527 (local address 127.12.2.65:39907)
0620 23:05:11.914901 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.914909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.915687 (+ 778us) server_negotiation.cc:407] Connection header received
0620 23:05:11.915816 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.915822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.915909 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.916004 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:11.916524 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.917648 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.920286 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.921508 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.921611 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.921703 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.922119 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.922245 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:05:11.922606 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.914596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39907 (local address 127.12.2.67:42527)
0620 23:05:11.915580 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:11.915596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:11.915707 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:11.916016 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:11.916020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:11.916042 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:11.916378 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.916393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.917799 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.917805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.920125 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.920140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.921683 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.921688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.921933 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.921958 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 23:05:11.922075 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:05:11.922177 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":174,"spinlock_wait_cycles":384,"thread_start_us":656,"threads_started":1}
I20260620 23:05:11.922665 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.914576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:38591 (local address 127.12.2.67:42703)
0620 23:05:11.915297 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:11.915319 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:11.915460 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:11.916024 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:11.916028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:11.916111 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:11.916486 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.916497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.917808 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.917811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.920127 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.920140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.921641 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.921647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.921849 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.921917 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 23:05:11.922013 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:05:11.922149 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":127,"spinlock_wait_cycles":26368,"thread_start_us":365,"threads_started":1}
I20260620 23:05:11.923296 14197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0d52a3c10d43bbb870a2d144b7f687" candidate_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" is_pre_election: true
I20260620 23:05:11.923659 14197 raft_consensus.cc:2454] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0860c2283ba84e0f80fcfa0a4efc7201 in term 0.
I20260620 23:05:11.924108 14401 leader_election.cc:302] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0860c2283ba84e0f80fcfa0a4efc7201, eaa332d5f0e64e03a248d2d2f2691b63; no voters:
I20260620 23:05:11.924064 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0d52a3c10d43bbb870a2d144b7f687" candidate_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" is_pre_election: true
I20260620 23:05:11.924433 14336 raft_consensus.cc:2454] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0860c2283ba84e0f80fcfa0a4efc7201 in term 0.
I20260620 23:05:11.924471 14545 raft_consensus.cc:2781] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:11.924573 14545 raft_consensus.cc:482] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:11.924657 14545 raft_consensus.cc:3037] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:11.926705 14545 raft_consensus.cc:504] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.927448 14545 leader_election.cc:288] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [CANDIDATE]: Term 1 election: Requested vote from peers eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65:39907), ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591)
I20260620 23:05:11.927858 14197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0d52a3c10d43bbb870a2d144b7f687" candidate_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa332d5f0e64e03a248d2d2f2691b63"
I20260620 23:05:11.927902 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0d52a3c10d43bbb870a2d144b7f687" candidate_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5"
I20260620 23:05:11.928061 14336 raft_consensus.cc:3037] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:11.928086 14197 raft_consensus.cc:3037] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:11.929934 14197 raft_consensus.cc:2454] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0860c2283ba84e0f80fcfa0a4efc7201 in term 1.
I20260620 23:05:11.929998 14336 raft_consensus.cc:2454] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0860c2283ba84e0f80fcfa0a4efc7201 in term 1.
I20260620 23:05:11.930315 14401 leader_election.cc:302] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0860c2283ba84e0f80fcfa0a4efc7201, eaa332d5f0e64e03a248d2d2f2691b63, ed74ae9cd6d4465c8a0166bf7bb459e5; no voters:
I20260620 23:05:11.930593 14545 raft_consensus.cc:2781] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:11.930996 14545 raft_consensus.cc:686] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 LEADER]: Becoming Leader. State: Replica: 0860c2283ba84e0f80fcfa0a4efc7201, State: Running, Role: LEADER
I20260620 23:05:11.931759 14545 consensus_queue.cc:237] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:11.935417 14564 tablet_replica.cc:439] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: TxnStatusTablet state changed. Reason: New leader 0860c2283ba84e0f80fcfa0a4efc7201. Latest consensus state: current_term: 1 leader_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } } }
I20260620 23:05:11.935750 14564 tablet_replica.cc:442] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:05:11.936354 14566 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:05:11.936609 14566 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:05:11.936821 14035 catalog_manager.cc:5495] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67). New cstate: current_term: 1 leader_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } health_report { overall_health: HEALTHY } } }
I20260620 23:05:11.975327 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.969428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:36543 (local address 127.0.0.1:36024)
0620 23:05:11.969772 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:11.969792 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:11.969954 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:11.970462 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:11.970466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:11.970480 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:11.970672 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.970681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.971785 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.971791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.972828 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:11.972841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.973563 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.973570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:11.973753 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.973811 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:11.974704 (+ 893us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:11.974729 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 23:05:11.974816 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:05:11.975021 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260620 23:05:11.975327 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:11.969568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.12.2.67:36543)
0620 23:05:11.969747 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:11.969754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:11.969865 (+ 111us) server_negotiation.cc:407] Connection header received
0620 23:05:11.970171 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:11.970176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:11.970290 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:11.970426 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:11.970805 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.971644 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.972974 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:11.973407 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:11.973543 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:11.974002 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:11.974589 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:11.974741 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:11.974835 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:05:11.974952 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:05:11.983454 14336 raft_consensus.cc:1261] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Refusing update from remote peer 0860c2283ba84e0f80fcfa0a4efc7201: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:11.983610 14197 raft_consensus.cc:1261] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Refusing update from remote peer 0860c2283ba84e0f80fcfa0a4efc7201: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:11.984458 14564 consensus_queue.cc:1035] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:11.984841 14545 consensus_queue.cc:1035] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:11.992342 14547 tablet_replica.cc:439] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63: TxnStatusTablet state changed. Reason: New leader 0860c2283ba84e0f80fcfa0a4efc7201. Latest consensus state: current_term: 1 leader_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } } }
I20260620 23:05:11.992638 14547 tablet_replica.cc:442] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:11.996095 14569 mvcc.cc:205] Tried to move back new op lower bound from 7299058532274163712 to 7299058532076658688. Current Snapshot: MvccSnapshot[applied={T|T < 7299058532274163712}]
I20260620 23:05:11.996615 14545 tablet_replica.cc:439] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } } }
I20260620 23:05:11.996906 14545 tablet_replica.cc:442] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:05:12.004657 14546 tablet_replica.cc:439] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5: TxnStatusTablet state changed. Reason: New leader 0860c2283ba84e0f80fcfa0a4efc7201. Latest consensus state: current_term: 1 leader_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } } }
I20260620 23:05:12.004980 14546 tablet_replica.cc:442] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:12.006448 14547 tablet_replica.cc:439] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } } }
I20260620 23:05:12.006687 14547 tablet_replica.cc:442] T 5c0d52a3c10d43bbb870a2d144b7f687 P eaa332d5f0e64e03a248d2d2f2691b63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:12.008752 14564 tablet_replica.cc:439] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } } }
I20260620 23:05:12.009006 14564 tablet_replica.cc:442] T 5c0d52a3c10d43bbb870a2d144b7f687 P 0860c2283ba84e0f80fcfa0a4efc7201: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:05:12.012713 14546 tablet_replica.cc:439] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } } }
I20260620 23:05:12.013007 14546 tablet_replica.cc:442] T 5c0d52a3c10d43bbb870a2d144b7f687 P ed74ae9cd6d4465c8a0166bf7bb459e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:12.017459 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.009057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.12.2.66:38591)
0620 23:05:12.009812 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:12.009819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:12.011043 (+ 1224us) server_negotiation.cc:407] Connection header received
0620 23:05:12.011362 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:12.011367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:12.011459 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:12.011579 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:12.012258 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.013449 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.015508 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.015891 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.015983 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.016461 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:12.016952 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:12.017067 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:12.017173 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 23:05:12.017236 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616}
I20260620 23:05:12.032720 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.027031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:36543 (local address 127.12.2.66:52171)
0620 23:05:12.027580 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:12.027594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:12.027703 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:12.028015 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:12.028017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:12.028026 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:12.028259 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.028266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.029584 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.029587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.030639 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.030650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.031690 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.031695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.031813 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.031861 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:12.032272 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:12.032281 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:05:12.032343 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:05:12.032438 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":291,"threads_started":1}
I20260620 23:05:12.032745 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.026994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:52171 (local address 127.12.2.67:36543)
0620 23:05:12.027141 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:12.027145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:12.027674 (+ 529us) server_negotiation.cc:407] Connection header received
0620 23:05:12.027810 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:12.027816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:12.027920 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:12.028012 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:12.028376 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.029473 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.030769 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.031519 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.031647 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.032042 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:12.032170 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:12.032268 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:12.032376 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 23:05:12.032457 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:05:12.043028 14470 raft_consensus.cc:1261] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Refusing update from remote peer ed74ae9cd6d4465c8a0166bf7bb459e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:12.043255 14197 raft_consensus.cc:1261] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Refusing update from remote peer ed74ae9cd6d4465c8a0166bf7bb459e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:12.043784 14550 consensus_queue.cc:1035] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:12.044170 14546 consensus_queue.cc:1035] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:12.049783 14570 mvcc.cc:205] Tried to move back new op lower bound from 7299058532521435136 to 7299058531022811136. Current Snapshot: MvccSnapshot[applied={T|T < 7299058532521435136}]
I20260620 23:05:12.086759 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.080232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:58949 (local address 127.12.2.66:38591)
0620 23:05:12.080393 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:12.080399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:12.081096 (+ 697us) server_negotiation.cc:407] Connection header received
0620 23:05:12.081256 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:12.081260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:12.081322 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:12.081389 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:12.081987 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.082872 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.085041 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.085423 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.085560 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.086022 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:12.086151 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:12.086241 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:12.086383 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 23:05:12.086460 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:05:12.087005 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.080274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:38591 (local address 127.12.2.67:58949)
0620 23:05:12.080980 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:12.080998 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:12.081147 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:12.081425 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:12.081430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:12.081443 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:12.081826 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.081839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.083002 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.083011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.084907 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.084922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.085604 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.085610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.085778 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.085844 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:12.086260 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:12.086271 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:05:12.086350 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:05:12.086489 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":365,"threads_started":1}
I20260620 23:05:12.125167 14035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:12.125460 14035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:12.129423 14035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4: Sending DeleteTablet for 3 replicas of tablet c7e0789de61a4d12b6ea2324411c2493
I20260620 23:05:12.130524 14176 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e0789de61a4d12b6ea2324411c2493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:12 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:52326
I20260620 23:05:12.130529 14313 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e0789de61a4d12b6ea2324411c2493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:12 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:40486
I20260620 23:05:12.131757 14450 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e0789de61a4d12b6ea2324411c2493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:12 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:36014
I20260620 23:05:12.138212 14035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:12.140187 14035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:12.143014 14592 tablet_replica.cc:330] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: stopping tablet replica
I20260620 23:05:12.143471 14592 raft_consensus.cc:2229] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:12.144119 14592 raft_consensus.cc:2258] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:12.146083 14592 ts_tablet_manager.cc:1905] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:12.151844 14175 tablet_service.cc:1467] Processing CreateTablet for tablet 4c581a4d0bb64847964770e50f719ceb (DEFAULT_TABLE table=test-table [id=a82168305a76441989c67ee42cd14980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:12.152364 14175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c581a4d0bb64847964770e50f719ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:12.153731 14592 ts_tablet_manager.cc:1918] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:12.153885 14592 log.cc:1196] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/c7e0789de61a4d12b6ea2324411c2493
I20260620 23:05:12.154353 14592 ts_tablet_manager.cc:1939] T c7e0789de61a4d12b6ea2324411c2493 P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting consensus metadata
I20260620 23:05:12.156388 14313 tablet_service.cc:1467] Processing CreateTablet for tablet 4c581a4d0bb64847964770e50f719ceb (DEFAULT_TABLE table=test-table [id=a82168305a76441989c67ee42cd14980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:12.156780 14313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c581a4d0bb64847964770e50f719ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:12.156708 14449 tablet_service.cc:1467] Processing CreateTablet for tablet 4c581a4d0bb64847964770e50f719ceb (DEFAULT_TABLE table=test-table [id=a82168305a76441989c67ee42cd14980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:12.157250 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c581a4d0bb64847964770e50f719ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:12.162660 14021 catalog_manager.cc:4841] TS ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591): tablet c7e0789de61a4d12b6ea2324411c2493 (table test-table [id=9057d69cc36a4104a83248fe20dea509]) successfully deleted
I20260620 23:05:12.163065 14539 tablet_bootstrap.cc:492] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5: Bootstrap starting.
I20260620 23:05:12.163735 14542 tablet_bootstrap.cc:492] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201: Bootstrap starting.
I20260620 23:05:12.164570 14539 tablet_bootstrap.cc:654] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:12.165787 14542 tablet_bootstrap.cc:654] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:12.167708 14540 tablet_bootstrap.cc:492] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63: Bootstrap starting.
I20260620 23:05:12.169397 14540 tablet_bootstrap.cc:654] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:12.174484 14539 tablet_bootstrap.cc:492] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5: No bootstrap required, opened a new log
I20260620 23:05:12.174650 14539 ts_tablet_manager.cc:1397] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260620 23:05:12.175525 14539 raft_consensus.cc:348] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:12.175779 14539 raft_consensus.cc:374] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:12.175872 14539 raft_consensus.cc:729] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed74ae9cd6d4465c8a0166bf7bb459e5, State: Initialized, Role: FOLLOWER
I20260620 23:05:12.176153 14539 consensus_queue.cc:260] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:12.176771 14539 ts_tablet_manager.cc:1428] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:05:12.177183 14540 tablet_bootstrap.cc:492] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63: No bootstrap required, opened a new log
I20260620 23:05:12.177353 14540 ts_tablet_manager.cc:1397] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 23:05:12.178108 14540 raft_consensus.cc:348] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:12.178347 14540 raft_consensus.cc:374] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:12.178427 14540 raft_consensus.cc:729] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa332d5f0e64e03a248d2d2f2691b63, State: Initialized, Role: FOLLOWER
I20260620 23:05:12.178704 14540 consensus_queue.cc:260] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:12.180116 14540 ts_tablet_manager.cc:1428] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:12.184664 14542 tablet_bootstrap.cc:492] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201: No bootstrap required, opened a new log
I20260620 23:05:12.184865 14542 ts_tablet_manager.cc:1397] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.000s
I20260620 23:05:12.185660 14542 raft_consensus.cc:348] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:12.185917 14542 raft_consensus.cc:374] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:12.185992 14542 raft_consensus.cc:729] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0860c2283ba84e0f80fcfa0a4efc7201, State: Initialized, Role: FOLLOWER
I20260620 23:05:12.186254 14542 consensus_queue.cc:260] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:12.189100 14542 ts_tablet_manager.cc:1428] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:05:12.193466 14594 tablet_replica.cc:330] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: stopping tablet replica
I20260620 23:05:12.193935 14594 raft_consensus.cc:2229] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:12.194291 14594 raft_consensus.cc:2258] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:12.195111 14593 tablet_replica.cc:330] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: stopping tablet replica
I20260620 23:05:12.195510 14593 raft_consensus.cc:2229] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:12.195859 14593 raft_consensus.cc:2258] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:12.197576 14594 ts_tablet_manager.cc:1905] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:12.197639 14593 ts_tablet_manager.cc:1905] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:12.205667 14593 ts_tablet_manager.cc:1918] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:12.205924 14593 log.cc:1196] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/c7e0789de61a4d12b6ea2324411c2493
I20260620 23:05:12.206236 14594 ts_tablet_manager.cc:1918] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:12.206411 14594 log.cc:1196] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/c7e0789de61a4d12b6ea2324411c2493
I20260620 23:05:12.206606 14593 ts_tablet_manager.cc:1939] T c7e0789de61a4d12b6ea2324411c2493 P eaa332d5f0e64e03a248d2d2f2691b63: Deleting consensus metadata
I20260620 23:05:12.206995 14594 ts_tablet_manager.cc:1939] T c7e0789de61a4d12b6ea2324411c2493 P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting consensus metadata
I20260620 23:05:12.208693 14020 catalog_manager.cc:4841] TS 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543): tablet c7e0789de61a4d12b6ea2324411c2493 (table test-table [id=9057d69cc36a4104a83248fe20dea509]) successfully deleted
I20260620 23:05:12.209692 14020 catalog_manager.cc:4841] TS eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65:39907): tablet c7e0789de61a4d12b6ea2324411c2493 (table test-table [id=9057d69cc36a4104a83248fe20dea509]) successfully deleted
I20260620 23:05:12.235188 14547 raft_consensus.cc:482] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:12.235450 14547 raft_consensus.cc:504] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:12.237228 14547 leader_election.cc:288] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591), 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543)
I20260620 23:05:12.255628 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.238185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:34287 (local address 127.12.2.67:36543)
0620 23:05:12.238366 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:12.238372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:12.247924 (+ 9552us) server_negotiation.cc:407] Connection header received
0620 23:05:12.248184 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:12.248189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:12.248260 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:12.248321 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:12.249345 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.250504 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.253038 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.254398 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.254488 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.254600 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:12.255227 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 23:05:12.255359 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:05:12.255668 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.237900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:59219 (local address 127.12.2.66:38591)
0620 23:05:12.238122 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:12.238129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:12.247523 (+ 9394us) server_negotiation.cc:407] Connection header received
0620 23:05:12.247767 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:12.247772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:12.247853 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:12.247940 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:12.249000 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.250098 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.252987 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.254418 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.254527 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.254651 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:12.255262 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 23:05:12.255397 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:05:12.256507 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.237704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:36543 (local address 127.12.2.65:34287)
0620 23:05:12.247840 (+ 10136us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:12.247856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:12.248018 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:12.248505 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:12.248511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:12.248542 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:12.249153 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.249167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.250728 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.250733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.252873 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.252888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.254800 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.254807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.255005 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.255083 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 23:05:12.255891 (+ 808us) client_negotiation.cc:240] Negotiation successful
0620 23:05:12.256029 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9969,"mutex_wait_us":207,"thread_start_us":9416,"threads_started":1}
I20260620 23:05:12.257097 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.237477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:38591 (local address 127.12.2.65:59219)
0620 23:05:12.247378 (+ 9901us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:12.247402 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:12.247574 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:12.248324 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:12.248329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:12.248357 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:12.248814 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.248828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.250319 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.250324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.252806 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.252822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.254859 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.254866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.255082 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.255113 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:05:12.256347 (+ 1234us) client_negotiation.cc:240] Negotiation successful
0620 23:05:12.256483 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9767,"mutex_wait_us":4,"thread_start_us":11270,"threads_started":1}
I20260620 23:05:12.257242 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c581a4d0bb64847964770e50f719ceb" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" is_pre_election: true
I20260620 23:05:12.257508 14470 raft_consensus.cc:2454] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 0.
I20260620 23:05:12.258023 14123 leader_election.cc:302] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0860c2283ba84e0f80fcfa0a4efc7201, eaa332d5f0e64e03a248d2d2f2691b63; no voters:
I20260620 23:05:12.258424 14547 raft_consensus.cc:2781] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:12.258544 14547 raft_consensus.cc:482] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:12.258626 14547 raft_consensus.cc:3037] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:12.258816 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c581a4d0bb64847964770e50f719ceb" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" is_pre_election: true
I20260620 23:05:12.259064 14336 raft_consensus.cc:2454] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 0.
I20260620 23:05:12.261756 14547 raft_consensus.cc:504] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:12.262390 14547 leader_election.cc:288] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 election: Requested vote from peers ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591), 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543)
I20260620 23:05:12.262841 14470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c581a4d0bb64847964770e50f719ceb" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0860c2283ba84e0f80fcfa0a4efc7201"
I20260620 23:05:12.263088 14470 raft_consensus.cc:3037] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:12.263072 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c581a4d0bb64847964770e50f719ceb" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5"
I20260620 23:05:12.263244 14336 raft_consensus.cc:3037] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:12.266100 14470 raft_consensus.cc:2454] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 1.
I20260620 23:05:12.266152 14336 raft_consensus.cc:2454] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 1.
I20260620 23:05:12.266501 14123 leader_election.cc:302] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0860c2283ba84e0f80fcfa0a4efc7201, eaa332d5f0e64e03a248d2d2f2691b63; no voters:
I20260620 23:05:12.266969 14547 raft_consensus.cc:2781] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:12.268195 14547 raft_consensus.cc:686] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 LEADER]: Becoming Leader. State: Replica: eaa332d5f0e64e03a248d2d2f2691b63, State: Running, Role: LEADER
I20260620 23:05:12.268603 14547 consensus_queue.cc:237] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:12.272275 14035 catalog_manager.cc:5495] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65). New cstate: current_term: 1 leader_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:12.315102 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.307357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.12.2.65:39907)
0620 23:05:12.307530 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:12.307536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:12.307986 (+ 450us) server_negotiation.cc:407] Connection header received
0620 23:05:12.308085 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:12.308090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:12.308193 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:12.308338 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:12.308898 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.309982 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.311906 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.312512 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.312633 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.313233 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:12.313978 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:12.314121 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:12.314706 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 23:05:12.314813 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:05:12.343786 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.337171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:58717 (local address 127.12.2.67:36543)
0620 23:05:12.337342 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:12.337348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:12.337480 (+ 132us) server_negotiation.cc:407] Connection header received
0620 23:05:12.337772 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:12.337778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:12.337868 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:12.337967 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:12.338488 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.339648 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.341632 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.342340 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.342472 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.343021 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:12.343175 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:12.343268 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:12.343422 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:05:12.343501 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:05:12.344188 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.336777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:36543 (local address 127.12.2.65:58717)
0620 23:05:12.337381 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:12.337400 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:12.337616 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:12.337984 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:12.337989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:12.338002 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:12.338323 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.338335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.339819 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.339824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.341468 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.341485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.342509 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.342515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.342691 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.342780 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:12.343289 (+ 509us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:12.343300 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:05:12.343391 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 23:05:12.343520 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":575,"threads_started":1}
I20260620 23:05:12.354866 14470 raft_consensus.cc:1261] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Refusing update from remote peer eaa332d5f0e64e03a248d2d2f2691b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:12.355630 14336 raft_consensus.cc:1261] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Refusing update from remote peer eaa332d5f0e64e03a248d2d2f2691b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:12.355640 14600 consensus_queue.cc:1035] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:12.356220 14547 consensus_queue.cc:1035] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:12.393340 14571 mvcc.cc:205] Tried to move back new op lower bound from 7299058533797801984 to 7299058533456449536. Current Snapshot: MvccSnapshot[applied={T|T < 7299058533797801984}]
I20260620 23:05:12.412309 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.403144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39907 (local address 127.12.2.67:34127)
0620 23:05:12.403339 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:12.403358 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:12.403465 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:12.404176 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:12.404181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:12.404194 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:12.404405 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.404415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.405766 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.405771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.407424 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:12.407438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.411074 (+ 3636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.411080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:12.411270 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.411318 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:12.411739 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:12.411749 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:05:12.411810 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:05:12.411954 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 23:05:12.415184 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:12.403630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:34127 (local address 127.12.2.65:39907)
0620 23:05:12.403803 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:12.403809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:12.403825 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:05:12.403879 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:12.403884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:12.403958 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:12.404036 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:12.404567 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.405620 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.410113 (+ 4493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:12.410829 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:12.410966 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:12.411484 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:12.411613 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:12.411690 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:12.414746 (+ 3056us) server_negotiation.cc:299] Negotiation successful
0620 23:05:12.414832 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 23:05:12.416556 14570 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:05:14.020063 14034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:14.020282 14034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:14.024061 14034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4: Sending DeleteTablet for 3 replicas of tablet 4c581a4d0bb64847964770e50f719ceb
I20260620 23:05:14.024777 14176 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c581a4d0bb64847964770e50f719ceb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:14 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:52326
I20260620 23:05:14.024859 14313 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c581a4d0bb64847964770e50f719ceb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:14 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:40486
I20260620 23:05:14.024897 14450 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c581a4d0bb64847964770e50f719ceb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:14 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:36014
I20260620 23:05:14.038038 14632 tablet_replica.cc:330] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201: stopping tablet replica
I20260620 23:05:14.038345 14630 tablet_replica.cc:330] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63: stopping tablet replica
I20260620 23:05:14.038348 14632 raft_consensus.cc:2229] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:14.038555 14630 raft_consensus.cc:2229] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:14.038646 14632 raft_consensus.cc:2258] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:14.039155 14630 raft_consensus.cc:2258] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:14.039235 14631 tablet_replica.cc:330] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5: stopping tablet replica
I20260620 23:05:14.039731 14632 ts_tablet_manager.cc:1905] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:14.039449 14631 raft_consensus.cc:2229] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:14.040191 14631 raft_consensus.cc:2258] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:14.040268 14630 ts_tablet_manager.cc:1905] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:14.041265 14631 ts_tablet_manager.cc:1905] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:14.046357 14632 ts_tablet_manager.cc:1918] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:05:14.046542 14632 log.cc:1196] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/4c581a4d0bb64847964770e50f719ceb
I20260620 23:05:14.046761 14630 ts_tablet_manager.cc:1918] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:05:14.047116 14632 ts_tablet_manager.cc:1939] T 4c581a4d0bb64847964770e50f719ceb P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting consensus metadata
I20260620 23:05:14.046978 14630 log.cc:1196] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/4c581a4d0bb64847964770e50f719ceb
I20260620 23:05:14.047791 14630 ts_tablet_manager.cc:1939] T 4c581a4d0bb64847964770e50f719ceb P eaa332d5f0e64e03a248d2d2f2691b63: Deleting consensus metadata
I20260620 23:05:14.047776 14631 ts_tablet_manager.cc:1918] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:05:14.047892 14631 log.cc:1196] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/4c581a4d0bb64847964770e50f719ceb
I20260620 23:05:14.048308 14020 catalog_manager.cc:4841] TS 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543): tablet 4c581a4d0bb64847964770e50f719ceb (table test-table [id=a82168305a76441989c67ee42cd14980]) successfully deleted
I20260620 23:05:14.048367 14631 ts_tablet_manager.cc:1939] T 4c581a4d0bb64847964770e50f719ceb P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting consensus metadata
I20260620 23:05:14.048995 14020 catalog_manager.cc:4841] TS eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65:39907): tablet 4c581a4d0bb64847964770e50f719ceb (table test-table [id=a82168305a76441989c67ee42cd14980]) successfully deleted
I20260620 23:05:14.049355 14021 catalog_manager.cc:4841] TS ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591): tablet 4c581a4d0bb64847964770e50f719ceb (table test-table [id=a82168305a76441989c67ee42cd14980]) successfully deleted
Jun 20 23:05:14 dist-test-slave-7g8g krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996714, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:05:14 dist-test-slave-7g8g krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996714, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:14.081580 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:14.063035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.12.2.126:41193)
0620 23:05:14.063565 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:14.063569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:14.063591 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:05:14.063728 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:14.063732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:14.063795 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:14.063882 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:14.065103 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.065830 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.066967 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.067378 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.067487 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:14.076685 (+ 9198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:14.076690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:14.076693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:14.076733 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:14.079134 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:14.079776 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:14.079780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:14.079782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:14.079897 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:14.080228 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:14.080233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:14.080236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:14.080525 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:14.080723 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:14.081030 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 23:05:14.081241 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":262,"threads_started":1}
I20260620 23:05:14.084892 14034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:14.085491 14034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:14.091492 14176 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec36d7d0f084376a03089659a97eecc (DEFAULT_TABLE table=test-table [id=ba5b6ded36e048f99fbf06972a76f4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:14.091658 14313 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec36d7d0f084376a03089659a97eecc (DEFAULT_TABLE table=test-table [id=ba5b6ded36e048f99fbf06972a76f4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:14.091786 14450 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec36d7d0f084376a03089659a97eecc (DEFAULT_TABLE table=test-table [id=ba5b6ded36e048f99fbf06972a76f4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:14.091907 14176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec36d7d0f084376a03089659a97eecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:14.092041 14313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec36d7d0f084376a03089659a97eecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:14.092151 14450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec36d7d0f084376a03089659a97eecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:14.096755 14641 tablet_bootstrap.cc:492] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5: Bootstrap starting.
I20260620 23:05:14.098260 14641 tablet_bootstrap.cc:654] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:14.099568 14643 tablet_bootstrap.cc:492] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201: Bootstrap starting.
I20260620 23:05:14.100941 14643 tablet_bootstrap.cc:654] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:14.101498 14641 tablet_bootstrap.cc:492] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5: No bootstrap required, opened a new log
I20260620 23:05:14.101611 14641 ts_tablet_manager.cc:1397] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 23:05:14.102200 14641 raft_consensus.cc:348] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.102375 14641 raft_consensus.cc:374] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:14.102455 14641 raft_consensus.cc:729] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed74ae9cd6d4465c8a0166bf7bb459e5, State: Initialized, Role: FOLLOWER
I20260620 23:05:14.102658 14641 consensus_queue.cc:260] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.104019 14641 ts_tablet_manager.cc:1428] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:05:14.105351 14643 tablet_bootstrap.cc:492] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201: No bootstrap required, opened a new log
I20260620 23:05:14.105517 14643 ts_tablet_manager.cc:1397] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260620 23:05:14.105885 14642 tablet_bootstrap.cc:492] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63: Bootstrap starting.
I20260620 23:05:14.106302 14643 raft_consensus.cc:348] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.106564 14643 raft_consensus.cc:374] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:14.106698 14643 raft_consensus.cc:729] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0860c2283ba84e0f80fcfa0a4efc7201, State: Initialized, Role: FOLLOWER
I20260620 23:05:14.106995 14643 consensus_queue.cc:260] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.107497 14642 tablet_bootstrap.cc:654] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:14.107542 14643 ts_tablet_manager.cc:1428] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:14.109397 14642 tablet_bootstrap.cc:492] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63: No bootstrap required, opened a new log
I20260620 23:05:14.109516 14642 ts_tablet_manager.cc:1397] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:05:14.110101 14642 raft_consensus.cc:348] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.110270 14642 raft_consensus.cc:374] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:14.110357 14642 raft_consensus.cc:729] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa332d5f0e64e03a248d2d2f2691b63, State: Initialized, Role: FOLLOWER
I20260620 23:05:14.110569 14642 consensus_queue.cc:260] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.110998 14642 ts_tablet_manager.cc:1428] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 23:05:14.262198 14621 raft_consensus.cc:482] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:14.262388 14621 raft_consensus.cc:504] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.263059 14621 leader_election.cc:288] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591), 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543)
I20260620 23:05:14.263585 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec36d7d0f084376a03089659a97eecc" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" is_pre_election: true
I20260620 23:05:14.263592 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec36d7d0f084376a03089659a97eecc" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" is_pre_election: true
I20260620 23:05:14.263792 14469 raft_consensus.cc:2454] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 0.
I20260620 23:05:14.263813 14336 raft_consensus.cc:2454] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 0.
I20260620 23:05:14.264219 14123 leader_election.cc:302] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0860c2283ba84e0f80fcfa0a4efc7201, eaa332d5f0e64e03a248d2d2f2691b63; no voters:
I20260620 23:05:14.264472 14621 raft_consensus.cc:2781] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:14.264565 14621 raft_consensus.cc:482] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:14.264626 14621 raft_consensus.cc:3037] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:14.266512 14621 raft_consensus.cc:504] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.266999 14621 leader_election.cc:288] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 election: Requested vote from peers ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591), 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543)
I20260620 23:05:14.267429 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec36d7d0f084376a03089659a97eecc" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5"
I20260620 23:05:14.267452 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec36d7d0f084376a03089659a97eecc" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0860c2283ba84e0f80fcfa0a4efc7201"
I20260620 23:05:14.267587 14336 raft_consensus.cc:3037] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:14.267611 14469 raft_consensus.cc:3037] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:14.269600 14469 raft_consensus.cc:2454] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 1.
I20260620 23:05:14.269766 14336 raft_consensus.cc:2454] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 1.
I20260620 23:05:14.269954 14123 leader_election.cc:302] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0860c2283ba84e0f80fcfa0a4efc7201, eaa332d5f0e64e03a248d2d2f2691b63; no voters:
I20260620 23:05:14.270201 14621 raft_consensus.cc:2781] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:14.270334 14621 raft_consensus.cc:686] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 LEADER]: Becoming Leader. State: Replica: eaa332d5f0e64e03a248d2d2f2691b63, State: Running, Role: LEADER
I20260620 23:05:14.270572 14621 consensus_queue.cc:237] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.273154 14036 catalog_manager.cc:5495] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65). New cstate: current_term: 1 leader_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:14.302179 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:14.297028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.12.2.65:39907)
0620 23:05:14.297823 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:14.297827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:14.297844 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:05:14.297890 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:14.297894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:14.297948 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:14.298030 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:14.298427 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.299287 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.300595 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.301013 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.301126 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:14.301500 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:14.301607 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:14.301680 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:14.301821 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 23:05:14.301893 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":561,"threads_started":1}
I20260620 23:05:14.310724 14469 raft_consensus.cc:1261] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Refusing update from remote peer eaa332d5f0e64e03a248d2d2f2691b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:14.311007 14336 raft_consensus.cc:1261] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Refusing update from remote peer eaa332d5f0e64e03a248d2d2f2691b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:14.311233 14621 consensus_queue.cc:1035] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:14.311599 14547 consensus_queue.cc:1035] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:14.352698 14034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:14.352906 14034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:14.355582 14034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4: Sending DeleteTablet for 3 replicas of tablet 3ec36d7d0f084376a03089659a97eecc
I20260620 23:05:14.356305 14176 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec36d7d0f084376a03089659a97eecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:14 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:52326
I20260620 23:05:14.356389 14313 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec36d7d0f084376a03089659a97eecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:14 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:40486
I20260620 23:05:14.356576 14630 tablet_replica.cc:330] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63: stopping tablet replica
I20260620 23:05:14.356555 14450 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec36d7d0f084376a03089659a97eecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:14 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:36014
I20260620 23:05:14.356621 14631 tablet_replica.cc:330] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5: stopping tablet replica
I20260620 23:05:14.356751 14630 raft_consensus.cc:2229] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:14.356792 14631 raft_consensus.cc:2229] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:14.356926 14632 tablet_replica.cc:330] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201: stopping tablet replica
I20260620 23:05:14.356961 14631 raft_consensus.cc:2258] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:14.357053 14630 raft_consensus.cc:2258] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:14.357163 14632 raft_consensus.cc:2229] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:14.357347 14632 raft_consensus.cc:2258] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:14.357342 14034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:14.357952 14034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:14.357978 14631 ts_tablet_manager.cc:1905] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:14.358376 14630 ts_tablet_manager.cc:1905] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:14.358604 14632 ts_tablet_manager.cc:1905] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:14.362866 14630 ts_tablet_manager.cc:1918] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:14.363019 14630 log.cc:1196] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/3ec36d7d0f084376a03089659a97eecc
I20260620 23:05:14.363399 14630 ts_tablet_manager.cc:1939] T 3ec36d7d0f084376a03089659a97eecc P eaa332d5f0e64e03a248d2d2f2691b63: Deleting consensus metadata
I20260620 23:05:14.364481 14631 ts_tablet_manager.cc:1918] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:14.364461 14020 catalog_manager.cc:4841] TS eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65:39907): tablet 3ec36d7d0f084376a03089659a97eecc (table test-table [id=ba5b6ded36e048f99fbf06972a76f4ea]) successfully deleted
I20260620 23:05:14.364614 14631 log.cc:1196] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/3ec36d7d0f084376a03089659a97eecc
I20260620 23:05:14.365025 14631 ts_tablet_manager.cc:1939] T 3ec36d7d0f084376a03089659a97eecc P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting consensus metadata
I20260620 23:05:14.365185 14632 ts_tablet_manager.cc:1918] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:14.365312 14632 log.cc:1196] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/3ec36d7d0f084376a03089659a97eecc
I20260620 23:05:14.365660 14632 ts_tablet_manager.cc:1939] T 3ec36d7d0f084376a03089659a97eecc P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting consensus metadata
I20260620 23:05:14.366062 14021 catalog_manager.cc:4841] TS ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591): tablet 3ec36d7d0f084376a03089659a97eecc (table test-table [id=ba5b6ded36e048f99fbf06972a76f4ea]) successfully deleted
I20260620 23:05:14.366550 14020 catalog_manager.cc:4841] TS 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543): tablet 3ec36d7d0f084376a03089659a97eecc (table test-table [id=ba5b6ded36e048f99fbf06972a76f4ea]) successfully deleted
I20260620 23:05:14.370733 14313 tablet_service.cc:1467] Processing CreateTablet for tablet fa4fe4c9072f47c6824b126db7790baf (DEFAULT_TABLE table=test-table [id=db018dbb1bd84bef851c452370ef573b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:14.370736 14176 tablet_service.cc:1467] Processing CreateTablet for tablet fa4fe4c9072f47c6824b126db7790baf (DEFAULT_TABLE table=test-table [id=db018dbb1bd84bef851c452370ef573b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:14.371146 14313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4fe4c9072f47c6824b126db7790baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:14.371160 14176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4fe4c9072f47c6824b126db7790baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:14.375909 14450 tablet_service.cc:1467] Processing CreateTablet for tablet fa4fe4c9072f47c6824b126db7790baf (DEFAULT_TABLE table=test-table [id=db018dbb1bd84bef851c452370ef573b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:14.376309 14450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4fe4c9072f47c6824b126db7790baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:14.376899 14641 tablet_bootstrap.cc:492] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5: Bootstrap starting.
I20260620 23:05:14.378754 14641 tablet_bootstrap.cc:654] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:14.381057 14642 tablet_bootstrap.cc:492] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63: Bootstrap starting.
I20260620 23:05:14.382118 14641 tablet_bootstrap.cc:492] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5: No bootstrap required, opened a new log
I20260620 23:05:14.382261 14641 ts_tablet_manager.cc:1397] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 23:05:14.382472 14642 tablet_bootstrap.cc:654] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:14.382876 14641 raft_consensus.cc:348] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.383140 14641 raft_consensus.cc:374] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:14.383206 14641 raft_consensus.cc:729] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed74ae9cd6d4465c8a0166bf7bb459e5, State: Initialized, Role: FOLLOWER
I20260620 23:05:14.383410 14641 consensus_queue.cc:260] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.383859 14641 ts_tablet_manager.cc:1428] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:14.387444 14643 tablet_bootstrap.cc:492] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201: Bootstrap starting.
I20260620 23:05:14.388918 14643 tablet_bootstrap.cc:654] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:14.390566 14642 tablet_bootstrap.cc:492] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63: No bootstrap required, opened a new log
I20260620 23:05:14.390709 14642 ts_tablet_manager.cc:1397] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260620 23:05:14.391108 14643 tablet_bootstrap.cc:492] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201: No bootstrap required, opened a new log
I20260620 23:05:14.391252 14643 ts_tablet_manager.cc:1397] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 23:05:14.391389 14642 raft_consensus.cc:348] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.391625 14642 raft_consensus.cc:374] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:14.391700 14642 raft_consensus.cc:729] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa332d5f0e64e03a248d2d2f2691b63, State: Initialized, Role: FOLLOWER
I20260620 23:05:14.391850 14643 raft_consensus.cc:348] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.391901 14642 consensus_queue.cc:260] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.392449 14642 ts_tablet_manager.cc:1428] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:14.392508 14643 raft_consensus.cc:374] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:14.392594 14643 raft_consensus.cc:729] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0860c2283ba84e0f80fcfa0a4efc7201, State: Initialized, Role: FOLLOWER
I20260620 23:05:14.392830 14643 consensus_queue.cc:260] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.393309 14643 ts_tablet_manager.cc:1428] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:14.532730 14624 raft_consensus.cc:482] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:14.532907 14624 raft_consensus.cc:504] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.533519 14624 leader_election.cc:288] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591), 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543)
I20260620 23:05:14.534137 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4fe4c9072f47c6824b126db7790baf" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" is_pre_election: true
I20260620 23:05:14.534188 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4fe4c9072f47c6824b126db7790baf" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" is_pre_election: true
I20260620 23:05:14.534395 14469 raft_consensus.cc:2454] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 0.
I20260620 23:05:14.534384 14336 raft_consensus.cc:2454] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 0.
I20260620 23:05:14.534802 14124 leader_election.cc:302] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaa332d5f0e64e03a248d2d2f2691b63, ed74ae9cd6d4465c8a0166bf7bb459e5; no voters:
I20260620 23:05:14.535132 14624 raft_consensus.cc:2781] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:14.535267 14624 raft_consensus.cc:482] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:14.535360 14624 raft_consensus.cc:3037] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:14.537438 14624 raft_consensus.cc:504] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.537914 14624 leader_election.cc:288] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 election: Requested vote from peers ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591), 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543)
I20260620 23:05:14.538326 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4fe4c9072f47c6824b126db7790baf" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5"
I20260620 23:05:14.538347 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4fe4c9072f47c6824b126db7790baf" candidate_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0860c2283ba84e0f80fcfa0a4efc7201"
I20260620 23:05:14.538499 14469 raft_consensus.cc:3037] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:14.538523 14336 raft_consensus.cc:3037] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:14.540419 14469 raft_consensus.cc:2454] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 1.
I20260620 23:05:14.540423 14336 raft_consensus.cc:2454] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaa332d5f0e64e03a248d2d2f2691b63 in term 1.
I20260620 23:05:14.540779 14123 leader_election.cc:302] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0860c2283ba84e0f80fcfa0a4efc7201, eaa332d5f0e64e03a248d2d2f2691b63; no voters:
I20260620 23:05:14.541064 14624 raft_consensus.cc:2781] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:14.541199 14624 raft_consensus.cc:686] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 LEADER]: Becoming Leader. State: Replica: eaa332d5f0e64e03a248d2d2f2691b63, State: Running, Role: LEADER
I20260620 23:05:14.541427 14624 consensus_queue.cc:237] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } }
I20260620 23:05:14.543741 14036 catalog_manager.cc:5495] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65). New cstate: current_term: 1 leader_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaa332d5f0e64e03a248d2d2f2691b63" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:14.584384 14336 raft_consensus.cc:1261] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Refusing update from remote peer eaa332d5f0e64e03a248d2d2f2691b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:14.584513 14469 raft_consensus.cc:1261] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Refusing update from remote peer eaa332d5f0e64e03a248d2d2f2691b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:14.584837 14547 consensus_queue.cc:1035] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed74ae9cd6d4465c8a0166bf7bb459e5" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:14.585063 14624 consensus_queue.cc:1035] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0860c2283ba84e0f80fcfa0a4efc7201" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:05:14.601245 14613 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:05:16.178637 14034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:16.178893 14034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:16.182809 14034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e576f91dd337497ab82b5d05473a69a4: Sending DeleteTablet for 3 replicas of tablet fa4fe4c9072f47c6824b126db7790baf
I20260620 23:05:16.183580 14176 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4fe4c9072f47c6824b126db7790baf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:16 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:52326
I20260620 23:05:16.183785 14313 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4fe4c9072f47c6824b126db7790baf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:16 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:40486
I20260620 23:05:16.183804 14450 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4fe4c9072f47c6824b126db7790baf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:16 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:36014
I20260620 23:05:16.184362 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14107
I20260620 23:05:16.196280 14683 tablet_replica.cc:330] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5: stopping tablet replica
I20260620 23:05:16.197211 14683 raft_consensus.cc:2229] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:16.197611 14683 raft_consensus.cc:2258] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:16.198724 14683 ts_tablet_manager.cc:1905] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:16.205482 14683 ts_tablet_manager.cc:1918] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:05:16.206087 14684 tablet_replica.cc:330] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201: stopping tablet replica
I20260620 23:05:16.206310 14684 raft_consensus.cc:2229] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:16.206540 14684 raft_consensus.cc:2258] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:16.206833 14683 log.cc:1196] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/fa4fe4c9072f47c6824b126db7790baf
I20260620 23:05:16.207306 14683 ts_tablet_manager.cc:1939] T fa4fe4c9072f47c6824b126db7790baf P ed74ae9cd6d4465c8a0166bf7bb459e5: Deleting consensus metadata
I20260620 23:05:16.208189 14684 ts_tablet_manager.cc:1905] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:16.208477 14021 catalog_manager.cc:4841] TS ed74ae9cd6d4465c8a0166bf7bb459e5 (127.12.2.66:38591): tablet fa4fe4c9072f47c6824b126db7790baf (table test-table [id=db018dbb1bd84bef851c452370ef573b]) successfully deleted
I20260620 23:05:16.210554 14682 tablet_replica.cc:330] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63: stopping tablet replica
I20260620 23:05:16.210760 14682 raft_consensus.cc:2229] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:16.211174 14682 raft_consensus.cc:2258] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:16.212069 14682 ts_tablet_manager.cc:1905] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:16.214221 14684 ts_tablet_manager.cc:1918] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:05:16.214345 14684 log.cc:1196] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/fa4fe4c9072f47c6824b126db7790baf
I20260620 23:05:16.214694 14684 ts_tablet_manager.cc:1939] T fa4fe4c9072f47c6824b126db7790baf P 0860c2283ba84e0f80fcfa0a4efc7201: Deleting consensus metadata
I20260620 23:05:16.215773 14020 catalog_manager.cc:4841] TS 0860c2283ba84e0f80fcfa0a4efc7201 (127.12.2.67:36543): tablet fa4fe4c9072f47c6824b126db7790baf (table test-table [id=db018dbb1bd84bef851c452370ef573b]) successfully deleted
I20260620 23:05:16.217453 14682 ts_tablet_manager.cc:1918] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:05:16.217566 14682 log.cc:1196] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/fa4fe4c9072f47c6824b126db7790baf
I20260620 23:05:16.217909 14682 ts_tablet_manager.cc:1939] T fa4fe4c9072f47c6824b126db7790baf P eaa332d5f0e64e03a248d2d2f2691b63: Deleting consensus metadata
Jun 20 23:05:16 dist-test-slave-7g8g krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996714, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:05:16.218799 14020 catalog_manager.cc:4841] TS eaa332d5f0e64e03a248d2d2f2691b63 (127.12.2.65:39907): tablet fa4fe4c9072f47c6824b126db7790baf (table test-table [id=db018dbb1bd84bef851c452370ef573b]) successfully deleted
I20260620 23:05:16.224766 14686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:16.186249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.12.2.65:39907)
0620 23:05:16.209807 (+ 23558us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:16.209811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:16.209830 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:05:16.209876 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:16.209880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:16.209941 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:16.210034 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:16.211751 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:16.212922 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:16.214907 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:16.215501 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:16.215630 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:16.219739 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:16.219770 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:16.219787 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:16.219819 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:16.222274 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:16.222881 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:16.222887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:16.222890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:16.223044 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:16.223386 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:16.223389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:16.223391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:16.223781 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:16.224032 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:16.224331 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 23:05:16.224497 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23447,"thread_start_us":23887,"threads_started":1}
W20260620 23:05:16.225419 14237 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52342
W20260620 23:05:16.226099 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14107: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:05:16.226737 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 14107
I20260620 23:05:16.244223 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14246
Jun 20 23:05:16 dist-test-slave-7g8g krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996714, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:05:16.258144 14687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:16.244916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.12.2.66:38591)
0620 23:05:16.245460 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:16.245464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:16.245478 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:05:16.245536 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:16.245540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:16.245594 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:16.245695 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:16.246824 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:16.247657 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:16.248859 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:16.249248 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:16.249347 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:16.253227 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:16.253240 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:16.253255 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:16.253289 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:16.255986 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:16.256554 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:16.256560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:16.256563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:16.256672 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:16.256953 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:16.256957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:16.256959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:16.257302 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:16.257503 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:16.257676 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 23:05:16.257835 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":302,"threads_started":1}
W20260620 23:05:16.258822 14376 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40494
W20260620 23:05:16.259377 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14246: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:05:16.259506 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 14246
I20260620 23:05:16.276489 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14385
Jun 20 23:05:16 dist-test-slave-7g8g krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996714, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:05:16.290642 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:16.277209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.12.2.67:36543)
0620 23:05:16.277693 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:16.277698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:16.277712 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:05:16.277760 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:16.277763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:16.277822 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:16.277911 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:16.279171 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:16.279944 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:16.281111 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:16.281520 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:16.281620 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:16.285474 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:16.285488 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:16.285505 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:16.285536 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:16.288174 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:16.288771 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:16.288777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:16.288779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:16.288902 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:16.289188 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:16.289192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:16.289195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:16.289602 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:16.289915 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:16.290065 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 23:05:16.290276 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":221,"threads_started":1}
W20260620 23:05:16.291400 14515 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36030
W20260620 23:05:16.291956 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14385: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:05:16.292094 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 14385
I20260620 23:05:16.308101 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14004
I20260620 23:05:16.309171 14076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:16.420164 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 14004
2026-06-20T23:05:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8192 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:16 dist-test-slave-7g8g krb5kdc[14692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:16 dist-test-slave-7g8g krb5kdc[14692](info): set up 2 sockets
Jun 20 23:05:16 dist-test-slave-7g8g krb5kdc[14692](info): commencing operation
krb5kdc: starting...
W20260620 23:05:17.410983 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:17 dist-test-slave-7g8g krb5kdc[14692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:05:17.584542 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:42713
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:42713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:17.889587 14705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:17.889864 14705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:17.889990 14705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:17.897046 14705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:17.897156 14705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:17.897233 14705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:17.897310 14705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:17.897365 14705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:17.907037 14705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:42713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:42713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:17.908346 14705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:17.909595 14705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:17.920940 14713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:17.920990 14711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:17.921063 14710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:17.921634 14705 server_base.cc:1034] running on GCE node
I20260620 23:05:17.922365 14705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:17.923710 14705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:17.924932 14705 hybrid_clock.cc:648] HybridClock initialized: now 1781996717924890 us; error 30 us; skew 500 ppm
I20260620 23:05:17.925252 14705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:17 dist-test-slave-7g8g krb5kdc[14692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996717, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:17.928071 14705 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:05:17.929479 14705 webserver.cc:460] Webserver started at http://127.12.2.126:34111/ using document root <none> and password file <none>
I20260620 23:05:17.930101 14705 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:17.930209 14705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:17.930488 14705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:17.933234 14705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "625dfc88a0ad480897a3b76a31bac1d4"
format_stamp: "Formatted at 2026-06-20 23:05:17 on dist-test-slave-7g8g"
server_key: "872bddff6372b64ae2d4f3cda574ba1e"
server_key_iv: "2ee36a6a4aaf968abdb8d76277772181"
server_key_version: "encryptionkey@0"
I20260620 23:05:17.933956 14705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "625dfc88a0ad480897a3b76a31bac1d4"
format_stamp: "Formatted at 2026-06-20 23:05:17 on dist-test-slave-7g8g"
server_key: "872bddff6372b64ae2d4f3cda574ba1e"
server_key_iv: "2ee36a6a4aaf968abdb8d76277772181"
server_key_version: "encryptionkey@0"
I20260620 23:05:17.938897 14705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:05:17.942176 14720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:17.943573 14705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:17.943748 14705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "625dfc88a0ad480897a3b76a31bac1d4"
format_stamp: "Formatted at 2026-06-20 23:05:17 on dist-test-slave-7g8g"
server_key: "872bddff6372b64ae2d4f3cda574ba1e"
server_key_iv: "2ee36a6a4aaf968abdb8d76277772181"
server_key_version: "encryptionkey@0"
I20260620 23:05:17.943945 14705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:17.960839 14705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:17.964721 14705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:17.964977 14705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:17.982159 14705 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:42713
I20260620 23:05:17.982180 14771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:42713 every 8 connection(s)
I20260620 23:05:17.983537 14705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:17.987082 14772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:17.990844 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 14705
I20260620 23:05:17.991060 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:17.991671 12297 external_mini_cluster.cc:1428] Setting key ad01f7d549589c60c8fed9e78f5e9034
I20260620 23:05:17.997619 14772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4: Bootstrap starting.
I20260620 23:05:18.001021 14772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:18.002126 14772 log.cc:824] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:18.005330 14772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4: No bootstrap required, opened a new log
Jun 20 23:05:18 dist-test-slave-7g8g krb5kdc[14692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:18.012893 14772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625dfc88a0ad480897a3b76a31bac1d4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42713 } }
I20260620 23:05:18.013247 14772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:18.013347 14772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625dfc88a0ad480897a3b76a31bac1d4, State: Initialized, Role: FOLLOWER
I20260620 23:05:18.013794 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:17.993537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.12.2.126:42713)
0620 23:05:17.994300 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:17.994315 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:17.994363 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:05:17.996137 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:17.996161 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:17.996557 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:17.996869 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:17.998586 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:17.999947 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:18.001628 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:18.002319 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:18.002460 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:18.006788 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:18.006796 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:18.006811 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:18.006851 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:18.009813 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:18.010455 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:18.010462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:18.010468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:18.010640 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:18.011009 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:18.011014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:18.011017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:18.011597 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:18.011879 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:18.012311 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 23:05:18.012716 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":329,"threads_started":1}
I20260620 23:05:18.013972 14772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625dfc88a0ad480897a3b76a31bac1d4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42713 } }
I20260620 23:05:18.014223 14772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:18.014369 14772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:18.014542 14772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:18.016891 14772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625dfc88a0ad480897a3b76a31bac1d4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42713 } }
I20260620 23:05:18.017343 14772 leader_election.cc:302] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 625dfc88a0ad480897a3b76a31bac1d4; no voters:
I20260620 23:05:18.017856 14772 leader_election.cc:288] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:18.017957 14777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:18.018528 14777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [term 1 LEADER]: Becoming Leader. State: Replica: 625dfc88a0ad480897a3b76a31bac1d4, State: Running, Role: LEADER
I20260620 23:05:18.019052 14777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625dfc88a0ad480897a3b76a31bac1d4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42713 } }
I20260620 23:05:18.019894 14772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:18.021674 14779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 625dfc88a0ad480897a3b76a31bac1d4. Latest consensus state: current_term: 1 leader_uuid: "625dfc88a0ad480897a3b76a31bac1d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625dfc88a0ad480897a3b76a31bac1d4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42713 } } }
I20260620 23:05:18.021736 14778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "625dfc88a0ad480897a3b76a31bac1d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625dfc88a0ad480897a3b76a31bac1d4" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42713 } } }
I20260620 23:05:18.022034 14779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:18.022037 14778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:18.023404 14782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:18.027437 14782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:18.035351 14782 catalog_manager.cc:1347] Generated new cluster ID: 67c72dedc47a454c9e11298eb1bd1900
I20260620 23:05:18.035451 14782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:18.045569 14782 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:18.046773 14782 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:18.053964 14782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 625dfc88a0ad480897a3b76a31bac1d4: Generated new TSK 0
I20260620 23:05:18.054770 14782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:05:18.102365 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42713
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:18.407855 14798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:18.408107 14798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:18.408208 14798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:18.415248 14798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:18.415361 14798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:18.415479 14798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:18.424937 14798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:18.426102 14798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:18.427315 14798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:18.439301 14806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:18.439352 14803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:18.439324 14804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:18.440062 14798 server_base.cc:1034] running on GCE node
I20260620 23:05:18.440595 14798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:18.441454 14798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:18.442649 14798 hybrid_clock.cc:648] HybridClock initialized: now 1781996718442585 us; error 56 us; skew 500 ppm
I20260620 23:05:18.442965 14798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:18 dist-test-slave-7g8g krb5kdc[14692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996718, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:18.445858 14798 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:05:18.447284 14798 webserver.cc:460] Webserver started at http://127.12.2.65:40111/ using document root <none> and password file <none>
I20260620 23:05:18.447930 14798 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:18.448020 14798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:18.448294 14798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:18.451068 14798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "f5d6144432a2498caa182dbbdb27669e"
format_stamp: "Formatted at 2026-06-20 23:05:18 on dist-test-slave-7g8g"
server_key: "4fa525f74f257a42e34b90dfae1c4019"
server_key_iv: "3ab6d49cbacaea6713ead3afc4433b4a"
server_key_version: "encryptionkey@0"
I20260620 23:05:18.451797 14798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "f5d6144432a2498caa182dbbdb27669e"
format_stamp: "Formatted at 2026-06-20 23:05:18 on dist-test-slave-7g8g"
server_key: "4fa525f74f257a42e34b90dfae1c4019"
server_key_iv: "3ab6d49cbacaea6713ead3afc4433b4a"
server_key_version: "encryptionkey@0"
I20260620 23:05:18.456563 14798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:05:18.459841 14813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:18.461104 14798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:05:18.461244 14798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "f5d6144432a2498caa182dbbdb27669e"
format_stamp: "Formatted at 2026-06-20 23:05:18 on dist-test-slave-7g8g"
server_key: "4fa525f74f257a42e34b90dfae1c4019"
server_key_iv: "3ab6d49cbacaea6713ead3afc4433b4a"
server_key_version: "encryptionkey@0"
I20260620 23:05:18.461416 14798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:18.493044 14798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:18.497179 14798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:18.497452 14798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:18.498448 14798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:18.499801 14798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:18.499886 14798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:18.500001 14798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:18.500063 14798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:18.528365 14798 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:33875
I20260620 23:05:18.528450 14925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:33875 every 8 connection(s)
I20260620 23:05:18.529585 14798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:18.539109 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 14798
I20260620 23:05:18.539326 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:18.540014 12297 external_mini_cluster.cc:1428] Setting key 658f0fdd650f5068c961baf584366a33
Jun 20 23:05:18 dist-test-slave-7g8g krb5kdc[14692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996718, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:18.548643 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:18.532064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:39477 (local address 127.12.2.126:42713)
0620 23:05:18.532509 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:18.532515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:18.533479 (+ 964us) server_negotiation.cc:407] Connection header received
0620 23:05:18.535218 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:18.535223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:18.535288 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:18.535378 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:18.536722 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:18.537431 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:18.538890 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:18.539478 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:18.539606 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:18.542921 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:18.542954 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:18.542959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:18.542999 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:18.545785 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:18.546435 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:18.546439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:18.546443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:18.546569 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:18.546906 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:18.546910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:18.546913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:18.547247 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:18.547356 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:18.548113 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 23:05:18.548272 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":218,"threads_started":1}
I20260620 23:05:18.549645 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:18.532622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:42713 (local address 127.12.2.65:39477)
0620 23:05:18.533334 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:18.533370 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:18.534958 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:18.535557 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:18.535564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:18.536037 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:18.536598 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:18.536614 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:18.537564 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:18.537568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:18.538749 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:18.538759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:18.539687 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:18.539693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:18.539894 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:18.540493 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:18.540513 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:18.542683 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:18.545924 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:18.545930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:18.545942 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:18.546329 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:18.546655 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:18.546658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:18.546660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:18.546824 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:18.547359 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:18.547367 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:18.547781 (+ 414us) client_negotiation.cc:769] Sending connection context
0620 23:05:18.548066 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 23:05:18.548561 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":289,"threads_started":1}
I20260620 23:05:18.551026 14926 heartbeater.cc:347] Connected to a master server at 127.12.2.126:42713
I20260620 23:05:18.551343 14926 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:18.552117 14926 heartbeater.cc:511] Master 127.12.2.126:42713 requested a full tablet report, sending...
I20260620 23:05:18.554390 14737 ts_manager.cc:194] Registered new tserver with Master: f5d6144432a2498caa182dbbdb27669e (127.12.2.65:33875)
I20260620 23:05:18.555830 14737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:39477
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:05:18.579847 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42713
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:18.881577 14932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:18.881872 14932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:18.881999 14932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:18.889070 14932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:18.889213 14932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:18.889374 14932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:18.899022 14932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:18.900377 14932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:18.901659 14932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:18.914032 14940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:18.914016 14938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:18.914532 14937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:18.914817 14932 server_base.cc:1034] running on GCE node
I20260620 23:05:18.915410 14932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:18.916321 14932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:18.917513 14932 hybrid_clock.cc:648] HybridClock initialized: now 1781996718917471 us; error 32 us; skew 500 ppm
I20260620 23:05:18.917809 14932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:18 dist-test-slave-7g8g krb5kdc[14692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996718, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:18.920878 14932 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:05:18.922338 14932 webserver.cc:460] Webserver started at http://127.12.2.66:43247/ using document root <none> and password file <none>
I20260620 23:05:18.923012 14932 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:18.923110 14932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:18.923362 14932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:18.926088 14932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "abf0c22e46ba4012aed25329775aadcb"
format_stamp: "Formatted at 2026-06-20 23:05:18 on dist-test-slave-7g8g"
server_key: "279e65088b3292c9f58acac0dc8a4b75"
server_key_iv: "36accb08e255db4c14fa43af1948c548"
server_key_version: "encryptionkey@0"
I20260620 23:05:18.926797 14932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "abf0c22e46ba4012aed25329775aadcb"
format_stamp: "Formatted at 2026-06-20 23:05:18 on dist-test-slave-7g8g"
server_key: "279e65088b3292c9f58acac0dc8a4b75"
server_key_iv: "36accb08e255db4c14fa43af1948c548"
server_key_version: "encryptionkey@0"
I20260620 23:05:18.931692 14932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 23:05:18.934906 14947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:18.936269 14932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:05:18.936404 14932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "abf0c22e46ba4012aed25329775aadcb"
format_stamp: "Formatted at 2026-06-20 23:05:18 on dist-test-slave-7g8g"
server_key: "279e65088b3292c9f58acac0dc8a4b75"
server_key_iv: "36accb08e255db4c14fa43af1948c548"
server_key_version: "encryptionkey@0"
I20260620 23:05:18.936602 14932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:18.980844 14932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:18.985015 14932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:18.985288 14932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:18.986207 14932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:18.987540 14932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:18.987614 14932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:18.987725 14932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:18.987779 14932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:19.017125 14932 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:38757
I20260620 23:05:19.017174 15059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:38757 every 8 connection(s)
I20260620 23:05:19.018332 14932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:19.025924 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 14932
I20260620 23:05:19.026118 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:19.026621 12297 external_mini_cluster.cc:1428] Setting key 0db44f22a118b8e3dfa0e0eaf6a0615f
Jun 20 23:05:19 dist-test-slave-7g8g krb5kdc[14692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996718, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:19.038452 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:19.020766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:56415 (local address 127.12.2.126:42713)
0620 23:05:19.020967 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:19.020973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:19.022186 (+ 1213us) server_negotiation.cc:407] Connection header received
0620 23:05:19.024020 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:19.024025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:19.024106 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:19.024183 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:19.025634 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:19.026698 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:19.028855 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:19.029436 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:19.029543 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:19.033077 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:19.033083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:19.033087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:19.033122 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:19.035687 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:19.036485 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:19.036489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:19.036491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:19.036607 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:19.037013 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:19.037016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:19.037018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:19.037233 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:19.037336 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:19.038049 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 23:05:19.038203 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:05:19.039891 15062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:19.021329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:42713 (local address 127.12.2.66:56415)
0620 23:05:19.022035 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:19.022077 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:19.023765 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:19.024419 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:19.024427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:19.024911 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:19.025489 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:19.025507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:19.026851 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:19.026855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:19.028707 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:19.028718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:19.029635 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:19.029640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:19.029839 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:19.030498 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:19.030520 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:19.032851 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:19.035823 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:19.035832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:19.035851 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:19.036385 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:19.036690 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:19.036695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:19.036698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:19.036913 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:19.037335 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:19.037343 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:19.037728 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 23:05:19.037992 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 23:05:19.038492 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":277,"threads_started":1}
I20260620 23:05:19.041280 15060 heartbeater.cc:347] Connected to a master server at 127.12.2.126:42713
I20260620 23:05:19.041592 15060 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:19.042241 15060 heartbeater.cc:511] Master 127.12.2.126:42713 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
I20260620 23:05:19.043614 14737 ts_manager.cc:194] Registered new tserver with Master: abf0c22e46ba4012aed25329775aadcb (127.12.2.66:38757)
I20260620 23:05:19.044420 14737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:56415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:05:19.066344 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42713
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:19.367540 15065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:19.367821 15065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:19.367939 15065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:19.375043 15065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:19.375146 15065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:19.375288 15065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:19.384796 15065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:19.385980 15065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:19.387280 15065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:19.402504 15071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:19.402756 15073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:19.402486 15070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:19.403260 15065 server_base.cc:1034] running on GCE node
I20260620 23:05:19.403864 15065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:19.404731 15065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:19.405917 15065 hybrid_clock.cc:648] HybridClock initialized: now 1781996719405867 us; error 40 us; skew 500 ppm
I20260620 23:05:19.406198 15065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:19 dist-test-slave-7g8g krb5kdc[14692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996719, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:19.412163 15065 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:05:19.413537 15065 webserver.cc:460] Webserver started at http://127.12.2.67:38753/ using document root <none> and password file <none>
I20260620 23:05:19.414168 15065 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:19.414271 15065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:19.414564 15065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:19.417374 15065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "fb444b7e96244599abab5751b7bdace1"
format_stamp: "Formatted at 2026-06-20 23:05:19 on dist-test-slave-7g8g"
server_key: "dbcecd0cb8dfaf79803edd3f1cc64995"
server_key_iv: "8631cdc1e3761f46403a7d98baa0af5a"
server_key_version: "encryptionkey@0"
I20260620 23:05:19.418097 15065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "fb444b7e96244599abab5751b7bdace1"
format_stamp: "Formatted at 2026-06-20 23:05:19 on dist-test-slave-7g8g"
server_key: "dbcecd0cb8dfaf79803edd3f1cc64995"
server_key_iv: "8631cdc1e3761f46403a7d98baa0af5a"
server_key_version: "encryptionkey@0"
I20260620 23:05:19.423220 15065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:05:19.426455 15080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:19.427734 15065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:19.427881 15065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "fb444b7e96244599abab5751b7bdace1"
format_stamp: "Formatted at 2026-06-20 23:05:19 on dist-test-slave-7g8g"
server_key: "dbcecd0cb8dfaf79803edd3f1cc64995"
server_key_iv: "8631cdc1e3761f46403a7d98baa0af5a"
server_key_version: "encryptionkey@0"
I20260620 23:05:19.428045 15065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:19.452713 15065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:19.456457 15065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:19.456707 15065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:19.457656 15065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:19.458956 15065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:19.459036 15065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:19.459118 15065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:19.459158 15065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:19.487553 15065 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:40463
I20260620 23:05:19.487632 15192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:40463 every 8 connection(s)
I20260620 23:05:19.488788 15065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:19.491254 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 15065
I20260620 23:05:19.491413 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:19.491876 12297 external_mini_cluster.cc:1428] Setting key f1e4e72692f58553aa14f71536ec63bf
Jun 20 23:05:19 dist-test-slave-7g8g krb5kdc[14692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996719, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:19.507949 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:19.491605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:54261 (local address 127.12.2.126:42713)
0620 23:05:19.491809 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:19.491816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:19.492683 (+ 867us) server_negotiation.cc:407] Connection header received
0620 23:05:19.494791 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:19.494796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:19.494902 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:19.495024 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:19.496563 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:19.497304 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:19.498725 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:19.499177 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:19.499264 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:19.502456 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:19.502462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:19.502465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:19.502501 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:19.504874 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:19.505565 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:19.505569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:19.505572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:19.505738 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:19.506248 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:19.506254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:19.506257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:19.506604 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:19.506734 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:19.507562 (+ 828us) server_negotiation.cc:299] Negotiation successful
0620 23:05:19.507718 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:05:19.508863 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:19.491775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:42713 (local address 127.12.2.67:54261)
0620 23:05:19.492523 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:19.492569 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:19.494505 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:19.495349 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:19.495358 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:19.495848 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:19.496411 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:19.496428 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:19.497416 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:19.497419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:19.498613 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:19.498624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:19.499338 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:19.499343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:19.499538 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:19.500097 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:19.500118 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:19.502246 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:19.505002 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:19.505008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:19.505020 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:19.505443 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:19.505852 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:19.505857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:19.505861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:19.506098 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:19.506733 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:19.506738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:19.507206 (+ 468us) client_negotiation.cc:769] Sending connection context
0620 23:05:19.507480 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 23:05:19.507845 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":296,"threads_started":1}
I20260620 23:05:19.510016 15193 heartbeater.cc:347] Connected to a master server at 127.12.2.126:42713
I20260620 23:05:19.510296 15193 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:19.510957 15193 heartbeater.cc:511] Master 127.12.2.126:42713 requested a full tablet report, sending...
I20260620 23:05:19.512224 14737 ts_manager.cc:194] Registered new tserver with Master: fb444b7e96244599abab5751b7bdace1 (127.12.2.67:40463)
I20260620 23:05:19.512966 14737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:54261
I20260620 23:05:19.519398 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:05:19.547030 14928 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:05:19.544737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.12.2.126:42713)
0620 23:05:19.544928 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:19.544933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:19.545334 (+ 401us) server_negotiation.cc:407] Connection header received
0620 23:05:19.545481 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:19.545485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:19.545547 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:19.545634 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:19.546725 (+ 1091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52000: BlockingRecv error: recv got EOF from 127.0.0.1:52000 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:05:19.550416 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14798
W20260620 23:05:19.552937 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14798: Not authorized: Client connection negotiation failed: client connection to 127.12.2.65:33875: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:05:19.553076 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 14798
I20260620 23:05:19.564847 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14932
W20260620 23:05:19.567184 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14932: Not authorized: Client connection negotiation failed: client connection to 127.12.2.66:38757: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:05:19.567396 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 14932
I20260620 23:05:19.578899 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15065
W20260620 23:05:19.581472 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15065: Not authorized: Client connection negotiation failed: client connection to 127.12.2.67:40463: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:05:19.581580 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 15065
I20260620 23:05:19.592685 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 14705
I20260620 23:05:19.593739 14767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:19.691941 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 14705
2026-06-20T23:05:19Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3269 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:19 dist-test-slave-7g8g krb5kdc[15210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:19 dist-test-slave-7g8g krb5kdc[15210](info): set up 2 sockets
Jun 20 23:05:19 dist-test-slave-7g8g krb5kdc[15210](info): commencing operation
krb5kdc: starting...
W20260620 23:05:20.678987 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:20 dist-test-slave-7g8g krb5kdc[15210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:05:20.845885 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:42055
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:42055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:21.151684 15223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:21.151961 15223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:21.152082 15223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:21.159153 15223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:21.159258 15223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:21.159328 15223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:21.159400 15223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:21.159458 15223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:21.169166 15223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:42055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:42055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:21.170416 15223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:21.171718 15223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:21.182451 15228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:21.182391 15229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:21.182510 15231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:21.183277 15223 server_base.cc:1034] running on GCE node
I20260620 23:05:21.183995 15223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:21.185302 15223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:21.186498 15223 hybrid_clock.cc:648] HybridClock initialized: now 1781996721186450 us; error 45 us; skew 500 ppm
I20260620 23:05:21.186822 15223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:21 dist-test-slave-7g8g krb5kdc[15210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:21.189757 15223 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:05:21.191242 15223 webserver.cc:460] Webserver started at http://127.12.2.126:32831/ using document root <none> and password file <none>
I20260620 23:05:21.191879 15223 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:21.191989 15223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:21.192291 15223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:21.195114 15223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "5e7a0b1b62804867b459810d7a15af5b"
format_stamp: "Formatted at 2026-06-20 23:05:21 on dist-test-slave-7g8g"
server_key: "2613a1cddfa0c4d694ebbbe9cd26fc30"
server_key_iv: "dbc7c295b734ec79cf9287db9bcc7d0d"
server_key_version: "encryptionkey@0"
I20260620 23:05:21.195829 15223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "5e7a0b1b62804867b459810d7a15af5b"
format_stamp: "Formatted at 2026-06-20 23:05:21 on dist-test-slave-7g8g"
server_key: "2613a1cddfa0c4d694ebbbe9cd26fc30"
server_key_iv: "dbc7c295b734ec79cf9287db9bcc7d0d"
server_key_version: "encryptionkey@0"
I20260620 23:05:21.200865 15223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:05:21.204360 15238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:21.205713 15223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:05:21.205886 15223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "5e7a0b1b62804867b459810d7a15af5b"
format_stamp: "Formatted at 2026-06-20 23:05:21 on dist-test-slave-7g8g"
server_key: "2613a1cddfa0c4d694ebbbe9cd26fc30"
server_key_iv: "dbc7c295b734ec79cf9287db9bcc7d0d"
server_key_version: "encryptionkey@0"
I20260620 23:05:21.206067 15223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:21.237746 15223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:21.241662 15223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:21.241902 15223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:21.258450 15223 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:42055
I20260620 23:05:21.258471 15289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:42055 every 8 connection(s)
I20260620 23:05:21.259759 15223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:21.261814 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 15223
I20260620 23:05:21.262017 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:21.262600 12297 external_mini_cluster.cc:1428] Setting key 0c398be7f58aeefcbec191c3e70cd61a
I20260620 23:05:21.263725 15290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:05:21 dist-test-slave-7g8g krb5kdc[15210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:21.276885 15290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b: Bootstrap starting.
I20260620 23:05:21.280027 15290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:21.281196 15290 log.cc:824] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:21.284318 15290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b: No bootstrap required, opened a new log
I20260620 23:05:21.284430 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:21.264531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.12.2.126:42055)
0620 23:05:21.265326 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:21.265341 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:21.265388 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:05:21.267101 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:21.267129 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:21.267648 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:21.268078 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:21.269302 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:21.271307 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:21.272503 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:21.273501 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:21.273685 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:21.277410 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:21.277419 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:21.277433 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:21.277469 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:21.280246 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:21.281014 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:21.281019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:21.281025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:21.281179 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:21.281558 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:21.281561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:21.281563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:21.282083 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:21.282333 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:21.282779 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 23:05:21.283216 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":551,"thread_start_us":322,"threads_started":1}
I20260620 23:05:21.288102 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:21.291934 15290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7a0b1b62804867b459810d7a15af5b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42055 } }
I20260620 23:05:21.292255 15290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:21.292339 15290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7a0b1b62804867b459810d7a15af5b, State: Initialized, Role: FOLLOWER
I20260620 23:05:21.292891 15290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7a0b1b62804867b459810d7a15af5b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42055 } }
I20260620 23:05:21.293097 15290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:21.293216 15290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:21.293362 15290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:21.295431 15290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7a0b1b62804867b459810d7a15af5b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42055 } }
I20260620 23:05:21.295964 15290 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e7a0b1b62804867b459810d7a15af5b; no voters:
I20260620 23:05:21.299155 15290 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:21.299242 15296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:21.301329 15290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:21.306082 15296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [term 1 LEADER]: Becoming Leader. State: Replica: 5e7a0b1b62804867b459810d7a15af5b, State: Running, Role: LEADER
I20260620 23:05:21.306638 15296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7a0b1b62804867b459810d7a15af5b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42055 } }
I20260620 23:05:21.309772 15297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e7a0b1b62804867b459810d7a15af5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7a0b1b62804867b459810d7a15af5b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42055 } } }
I20260620 23:05:21.310240 15297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:21.311163 15298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7a0b1b62804867b459810d7a15af5b. Latest consensus state: current_term: 1 leader_uuid: "5e7a0b1b62804867b459810d7a15af5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7a0b1b62804867b459810d7a15af5b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42055 } } }
I20260620 23:05:21.311510 15298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b [sys.catalog]: This master's current role is: LEADER
W20260620 23:05:21.317101 15312 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:05:21.317219 15312 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:05:21.317433 15311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:21.319526 15311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:21.328960 15311 catalog_manager.cc:1347] Generated new cluster ID: 0addb68fc8d34ddcac940b76641ef7e5
I20260620 23:05:21.329071 15311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:21.331218 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:21.323657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.12.2.126:42055)
0620 23:05:21.324007 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:21.324014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:21.324357 (+ 343us) server_negotiation.cc:407] Connection header received
0620 23:05:21.324529 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:21.324533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:21.324614 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:21.324728 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:21.325915 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:21.327004 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:21.328257 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:21.328671 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:21.328772 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:21.329817 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:21.329822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:21.329829 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:21.330423 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:21.330469 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:21.330490 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:21.330581 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:21.330710 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 23:05:21.330811 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260620 23:05:21.384467 15311 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:21.386107 15311 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:21.392448 15311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e7a0b1b62804867b459810d7a15af5b: Generated new TSK 0
I20260620 23:05:21.393186 15311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:21.403291 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15223
I20260620 23:05:21.404500 15285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:21.495445 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 15223
2026-06-20T23:05:21Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1799 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:21 dist-test-slave-7g8g krb5kdc[15324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:21 dist-test-slave-7g8g krb5kdc[15324](info): set up 2 sockets
Jun 20 23:05:21 dist-test-slave-7g8g krb5kdc[15324](info): commencing operation
krb5kdc: starting...
W20260620 23:05:22.475075 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:22 dist-test-slave-7g8g krb5kdc[15324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:05:22.645563 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:37607
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:37607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:22.949707 15337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:22.949995 15337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:22.950129 15337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:22.957283 15337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:22.957435 15337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:22.957517 15337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:22.957592 15337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:22.957651 15337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:22.967106 15337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:37607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:37607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:22.968303 15337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:22.969519 15337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:22.980059 15342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:22.980050 15345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:22.980156 15343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:22.981151 15337 server_base.cc:1034] running on GCE node
I20260620 23:05:22.981879 15337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:22.983245 15337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:22.984436 15337 hybrid_clock.cc:648] HybridClock initialized: now 1781996722984383 us; error 44 us; skew 500 ppm
I20260620 23:05:22.984742 15337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:22 dist-test-slave-7g8g krb5kdc[15324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996722, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:22.987685 15337 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:05:22.989125 15337 webserver.cc:460] Webserver started at http://127.12.2.126:33785/ using document root <none> and password file <none>
I20260620 23:05:22.989755 15337 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:22.989872 15337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:22.990145 15337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:22.992964 15337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "5247309a7dbb47de899ed01d49bb92b9"
format_stamp: "Formatted at 2026-06-20 23:05:22 on dist-test-slave-7g8g"
server_key: "20a0dc77d7fcda8f9fc7ee31356c6a3b"
server_key_iv: "debbc39a96261aca538ff366de84d434"
server_key_version: "encryptionkey@0"
I20260620 23:05:22.993650 15337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "5247309a7dbb47de899ed01d49bb92b9"
format_stamp: "Formatted at 2026-06-20 23:05:22 on dist-test-slave-7g8g"
server_key: "20a0dc77d7fcda8f9fc7ee31356c6a3b"
server_key_iv: "debbc39a96261aca538ff366de84d434"
server_key_version: "encryptionkey@0"
I20260620 23:05:22.998425 15337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260620 23:05:23.001698 15352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:23.003104 15337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:23.003265 15337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "5247309a7dbb47de899ed01d49bb92b9"
format_stamp: "Formatted at 2026-06-20 23:05:22 on dist-test-slave-7g8g"
server_key: "20a0dc77d7fcda8f9fc7ee31356c6a3b"
server_key_iv: "debbc39a96261aca538ff366de84d434"
server_key_version: "encryptionkey@0"
I20260620 23:05:23.003476 15337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:23.034160 15337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:23.038044 15337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:23.038298 15337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:23.053970 15337 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:37607
I20260620 23:05:23.053989 15403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:37607 every 8 connection(s)
I20260620 23:05:23.055541 15337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:23.059217 15404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:23.061637 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 15337
I20260620 23:05:23.061842 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:23.062570 12297 external_mini_cluster.cc:1428] Setting key 0a8af65dfdd6f0a5b5edc41b1f464011
I20260620 23:05:23.071046 15404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9: Bootstrap starting.
I20260620 23:05:23.074712 15404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:23.076025 15404 log.cc:824] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:05:23 dist-test-slave-7g8g krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:23.079051 15404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9: No bootstrap required, opened a new log
I20260620 23:05:23.084996 15407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:23.064615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.12.2.126:37607)
0620 23:05:23.065338 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:23.065349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:23.065390 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:05:23.066997 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:23.067018 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:23.067399 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:23.067723 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:23.069303 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:23.070586 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:23.072228 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:23.072921 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:23.073055 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:23.077680 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:23.077689 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:23.077704 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:23.077741 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:23.080945 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:23.081796 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:23.081803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:23.081809 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:23.081997 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:23.082364 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:23.082367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:23.082370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:23.082888 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:23.083160 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:23.083580 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 23:05:23.083966 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":345,"threads_started":1}
I20260620 23:05:23.087705 15404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5247309a7dbb47de899ed01d49bb92b9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37607 } }
I20260620 23:05:23.088131 15404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:23.088229 15404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5247309a7dbb47de899ed01d49bb92b9, State: Initialized, Role: FOLLOWER
I20260620 23:05:23.088784 15404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5247309a7dbb47de899ed01d49bb92b9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37607 } }
I20260620 23:05:23.088986 15404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:23.089107 15404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:23.089237 15404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:23.091535 15404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5247309a7dbb47de899ed01d49bb92b9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37607 } }
I20260620 23:05:23.091980 15404 leader_election.cc:302] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5247309a7dbb47de899ed01d49bb92b9; no voters:
I20260620 23:05:23.092538 15404 leader_election.cc:288] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:23.092653 15409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:23.093575 15409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [term 1 LEADER]: Becoming Leader. State: Replica: 5247309a7dbb47de899ed01d49bb92b9, State: Running, Role: LEADER
I20260620 23:05:23.094281 15409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5247309a7dbb47de899ed01d49bb92b9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37607 } }
I20260620 23:05:23.094506 15404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:23.097083 15411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5247309a7dbb47de899ed01d49bb92b9. Latest consensus state: current_term: 1 leader_uuid: "5247309a7dbb47de899ed01d49bb92b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5247309a7dbb47de899ed01d49bb92b9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37607 } } }
I20260620 23:05:23.097059 15410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5247309a7dbb47de899ed01d49bb92b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5247309a7dbb47de899ed01d49bb92b9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37607 } } }
I20260620 23:05:23.097438 15411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:23.097438 15410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:23.098215 15417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:23.103662 15417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:23.113415 15417 catalog_manager.cc:1347] Generated new cluster ID: ae7a2ac15603418b8e4c987bddc97186
I20260620 23:05:23.113526 15417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:23.122078 15417 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:23.123927 15417 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:23.131229 15417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5247309a7dbb47de899ed01d49bb92b9: Generated new TSK 0
I20260620 23:05:23.132243 15417 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:05:23.178429 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37607
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:23.497115 15430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:23.497381 15430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:23.497531 15430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:23.504602 15430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:23.504716 15430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:23.504884 15430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:23.514356 15430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:23.515617 15430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:23.516875 15430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:23.528718 15435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:23.528784 15436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:23.528930 15438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:23.529556 15430 server_base.cc:1034] running on GCE node
I20260620 23:05:23.530121 15430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:23.531001 15430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:23.532188 15430 hybrid_clock.cc:648] HybridClock initialized: now 1781996723532141 us; error 38 us; skew 500 ppm
I20260620 23:05:23.532478 15430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:23 dist-test-slave-7g8g krb5kdc[15324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996723, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:23.535570 15430 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:05:23.536898 15430 webserver.cc:460] Webserver started at http://127.12.2.65:41655/ using document root <none> and password file <none>
I20260620 23:05:23.537554 15430 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:23.537667 15430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:23.537930 15430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:23.540709 15430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "ce6eb3fdb4444f668277625f8a7b864e"
format_stamp: "Formatted at 2026-06-20 23:05:23 on dist-test-slave-7g8g"
server_key: "c8d705ccf044bdec58641fe75ef08692"
server_key_iv: "979a8150da20eec3e10493ce2f0f6a39"
server_key_version: "encryptionkey@0"
I20260620 23:05:23.541451 15430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "ce6eb3fdb4444f668277625f8a7b864e"
format_stamp: "Formatted at 2026-06-20 23:05:23 on dist-test-slave-7g8g"
server_key: "c8d705ccf044bdec58641fe75ef08692"
server_key_iv: "979a8150da20eec3e10493ce2f0f6a39"
server_key_version: "encryptionkey@0"
I20260620 23:05:23.546432 15430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:05:23.549572 15445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:23.550773 15430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:23.550913 15430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "ce6eb3fdb4444f668277625f8a7b864e"
format_stamp: "Formatted at 2026-06-20 23:05:23 on dist-test-slave-7g8g"
server_key: "c8d705ccf044bdec58641fe75ef08692"
server_key_iv: "979a8150da20eec3e10493ce2f0f6a39"
server_key_version: "encryptionkey@0"
I20260620 23:05:23.551129 15430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:23.561513 15430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:23.565083 15430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:23.565320 15430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:23.566150 15430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:23.567396 15430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:23.567467 15430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:23.567577 15430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:23.567627 15430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:23.596244 15430 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:45297
I20260620 23:05:23.596300 15557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:45297 every 8 connection(s)
I20260620 23:05:23.597455 15430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:23.606801 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 15430
I20260620 23:05:23.607015 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:23.607602 12297 external_mini_cluster.cc:1428] Setting key e2fd2fe6da6e97c6724e35cd74daacb8
Jun 20 23:05:23 dist-test-slave-7g8g krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996723, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:23.615447 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:23.599819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:40749 (local address 127.12.2.126:37607)
0620 23:05:23.600196 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:23.600200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:23.601038 (+ 838us) server_negotiation.cc:407] Connection header received
0620 23:05:23.602721 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:23.602727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:23.602801 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:23.602897 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:23.604288 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:23.604972 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:23.606403 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:23.606841 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:23.606967 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:23.610326 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:23.610331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:23.610334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:23.610372 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:23.612697 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:23.613339 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:23.613343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:23.613345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:23.613472 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:23.613848 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:23.613852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:23.613855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:23.614104 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:23.614241 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:23.614871 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 23:05:23.615096 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":178,"threads_started":1}
I20260620 23:05:23.616333 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:23.600248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37607 (local address 127.12.2.65:40749)
0620 23:05:23.600896 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:23.600937 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:23.602479 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:23.603140 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:23.603148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:23.603605 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:23.604136 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:23.604152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:23.605108 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:23.605111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:23.606268 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:23.606279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:23.607054 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:23.607061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:23.607321 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:23.607931 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:23.607959 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:23.610127 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:23.612817 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:23.612825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:23.612841 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:23.613232 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:23.613576 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:23.613581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:23.613584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:23.613751 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:23.614228 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:23.614235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:23.614545 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 23:05:23.614777 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 23:05:23.615296 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":248,"threads_started":1}
I20260620 23:05:23.617784 15558 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37607
I20260620 23:05:23.618059 15558 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:23.618692 15558 heartbeater.cc:511] Master 127.12.2.126:37607 requested a full tablet report, sending...
I20260620 23:05:23.620702 15369 ts_manager.cc:194] Registered new tserver with Master: ce6eb3fdb4444f668277625f8a7b864e (127.12.2.65:45297)
I20260620 23:05:23.622061 15369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:40749
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:05:23.645046 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37607
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:23.943045 15564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:23.943301 15564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:23.943435 15564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:23.950342 15564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:23.950449 15564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:23.950603 15564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:23.959991 15564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:23.961167 15564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:23.962390 15564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:23.974334 15569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:23.974388 15570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:23.974632 15572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:23.975325 15564 server_base.cc:1034] running on GCE node
I20260620 23:05:23.975886 15564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:23.976768 15564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:23.977969 15564 hybrid_clock.cc:648] HybridClock initialized: now 1781996723977912 us; error 46 us; skew 500 ppm
I20260620 23:05:23.978261 15564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:23 dist-test-slave-7g8g krb5kdc[15324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996723, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:23.981065 15564 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:05:23.982365 15564 webserver.cc:460] Webserver started at http://127.12.2.66:34343/ using document root <none> and password file <none>
I20260620 23:05:23.983060 15564 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:23.983177 15564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:23.983448 15564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:23.986193 15564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "de16bcdc8b5640b6b93e6367dc6edb2d"
format_stamp: "Formatted at 2026-06-20 23:05:23 on dist-test-slave-7g8g"
server_key: "18dfcd582fb9ad67340de3fbc476f4b9"
server_key_iv: "c69a42f80d8d678814803eb8b99a52a7"
server_key_version: "encryptionkey@0"
I20260620 23:05:23.986981 15564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "de16bcdc8b5640b6b93e6367dc6edb2d"
format_stamp: "Formatted at 2026-06-20 23:05:23 on dist-test-slave-7g8g"
server_key: "18dfcd582fb9ad67340de3fbc476f4b9"
server_key_iv: "c69a42f80d8d678814803eb8b99a52a7"
server_key_version: "encryptionkey@0"
I20260620 23:05:23.992172 15564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:05:23.995432 15579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:23.996733 15564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:05:23.996876 15564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "de16bcdc8b5640b6b93e6367dc6edb2d"
format_stamp: "Formatted at 2026-06-20 23:05:23 on dist-test-slave-7g8g"
server_key: "18dfcd582fb9ad67340de3fbc476f4b9"
server_key_iv: "c69a42f80d8d678814803eb8b99a52a7"
server_key_version: "encryptionkey@0"
I20260620 23:05:23.997040 15564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:24.024015 15564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:24.027920 15564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:24.028170 15564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:24.029139 15564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:24.030360 15564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:24.030442 15564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:24.030557 15564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:24.030608 15564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:24.059813 15564 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:45131
I20260620 23:05:24.059829 15691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:45131 every 8 connection(s)
I20260620 23:05:24.061313 15564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:24.069798 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 15564
I20260620 23:05:24.069957 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:24.070510 12297 external_mini_cluster.cc:1428] Setting key 32f5e7720593874d1e27c9d1ee5cde93
Jun 20 23:05:24 dist-test-slave-7g8g krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996723, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:24.080955 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:24.063805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:45763 (local address 127.12.2.126:37607)
0620 23:05:24.063958 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:24.063963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:24.065235 (+ 1272us) server_negotiation.cc:407] Connection header received
0620 23:05:24.067016 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:24.067022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:24.067116 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:24.067200 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:24.068641 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.069344 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.070926 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.071495 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.071610 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:24.075376 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:24.075381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:24.075385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:24.075416 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:24.078062 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.078787 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.078793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.078797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.078986 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.079352 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.079357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.079360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.079682 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:24.079803 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:24.080407 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 23:05:24.080628 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:05:24.081789 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:24.064342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37607 (local address 127.12.2.66:45763)
0620 23:05:24.065086 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:24.065121 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:24.066738 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:24.067436 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:24.067444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:24.067920 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:24.068465 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:24.068481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.069486 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.069490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:24.070801 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:24.070812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.071724 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.071731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:24.072022 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:24.072643 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:24.072669 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:24.075181 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:24.078203 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:24.078210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:24.078223 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:24.078668 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:24.079085 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:24.079089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:24.079091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:24.079263 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:24.079782 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:24.079788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:24.080095 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 23:05:24.080313 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 23:05:24.080697 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":275,"threads_started":1}
I20260620 23:05:24.083436 15692 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37607
I20260620 23:05:24.083777 15692 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:24.084529 15692 heartbeater.cc:511] Master 127.12.2.126:37607 requested a full tablet report, sending...
I20260620 23:05:24.085896 15369 ts_manager.cc:194] Registered new tserver with Master: de16bcdc8b5640b6b93e6367dc6edb2d (127.12.2.66:45131)
I20260620 23:05:24.086663 15369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:45763
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:05:24.114593 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37607
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:24.421938 15697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:24.422222 15697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:24.422374 15697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:24.429896 15697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:24.430013 15697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:24.430162 15697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:24.439785 15697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:24.441058 15697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:24.442322 15697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:24.454329 15703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:24.454632 15702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:24.454664 15705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:24.455202 15697 server_base.cc:1034] running on GCE node
I20260620 23:05:24.455854 15697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:24.456732 15697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:24.457923 15697 hybrid_clock.cc:648] HybridClock initialized: now 1781996724457879 us; error 34 us; skew 500 ppm
I20260620 23:05:24.458197 15697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:24 dist-test-slave-7g8g krb5kdc[15324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996724, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:24.464128 15697 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:05:24.465418 15697 webserver.cc:460] Webserver started at http://127.12.2.67:44619/ using document root <none> and password file <none>
I20260620 23:05:24.466027 15697 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:24.466116 15697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:24.466348 15697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:24.469028 15697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "61be63efbf8143df88b7d044c758d8c5"
format_stamp: "Formatted at 2026-06-20 23:05:24 on dist-test-slave-7g8g"
server_key: "610d0700a282cdfc2404cb28938e6feb"
server_key_iv: "11112e7ea22fda0f32e8e54347e4974d"
server_key_version: "encryptionkey@0"
I20260620 23:05:24.469700 15697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "61be63efbf8143df88b7d044c758d8c5"
format_stamp: "Formatted at 2026-06-20 23:05:24 on dist-test-slave-7g8g"
server_key: "610d0700a282cdfc2404cb28938e6feb"
server_key_iv: "11112e7ea22fda0f32e8e54347e4974d"
server_key_version: "encryptionkey@0"
I20260620 23:05:24.474210 15697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260620 23:05:24.477269 15712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:24.478490 15697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:24.478636 15697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "61be63efbf8143df88b7d044c758d8c5"
format_stamp: "Formatted at 2026-06-20 23:05:24 on dist-test-slave-7g8g"
server_key: "610d0700a282cdfc2404cb28938e6feb"
server_key_iv: "11112e7ea22fda0f32e8e54347e4974d"
server_key_version: "encryptionkey@0"
I20260620 23:05:24.478793 15697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:24.500384 15697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:24.504359 15697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:24.504607 15697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:24.505470 15697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:24.506706 15697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:24.506783 15697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:24.506903 15697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:24.506976 15697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 23:05:24.536453 15697 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:35019
I20260620 23:05:24.536505 15824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:35019 every 8 connection(s)
I20260620 23:05:24.537657 15697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:24.540474 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 15697
I20260620 23:05:24.540627 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:24.541054 12297 external_mini_cluster.cc:1428] Setting key 4b272d2a88a8e7d60e2ee102b9a445c1
Jun 20 23:05:24 dist-test-slave-7g8g krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996724, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:24.556493 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:24.540072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:59223 (local address 127.12.2.126:37607)
0620 23:05:24.540271 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:24.540277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:24.541507 (+ 1230us) server_negotiation.cc:407] Connection header received
0620 23:05:24.543391 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:24.543396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:24.543476 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:24.543573 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:24.544988 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.545728 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.547280 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.547726 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.547814 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:24.550909 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:24.550914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:24.550917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:24.550983 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:24.553352 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.554110 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.554114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.554116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.554283 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.554799 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.554804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.554807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.555174 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:24.555312 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:24.556077 (+ 765us) server_negotiation.cc:299] Negotiation successful
0620 23:05:24.556227 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:05:24.557305 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:24.540548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37607 (local address 127.12.2.67:59223)
0620 23:05:24.541314 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:24.541403 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:24.543148 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:24.543808 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:24.543819 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:24.544291 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:24.544823 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:24.544843 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.545882 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.545886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:24.547147 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:24.547159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.547910 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.547914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:24.548103 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:24.548692 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:24.548712 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:24.550707 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:24.553481 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:24.553489 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:24.553505 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:24.553990 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:24.554392 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:24.554397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:24.554400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:24.554681 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:24.555287 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:24.555295 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:24.555647 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 23:05:24.555921 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 23:05:24.556287 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":294,"threads_started":1}
I20260620 23:05:24.558512 15825 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37607
I20260620 23:05:24.558785 15825 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:24.559520 15825 heartbeater.cc:511] Master 127.12.2.126:37607 requested a full tablet report, sending...
I20260620 23:05:24.560811 15369 ts_manager.cc:194] Registered new tserver with Master: 61be63efbf8143df88b7d044c758d8c5 (127.12.2.67:35019)
I20260620 23:05:24.561525 15369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:59223
I20260620 23:05:24.567670 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:05:24 dist-test-slave-7g8g krb5kdc[15324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996724, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 23:05:24 dist-test-slave-7g8g krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996724, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:24.610195 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:24.597327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.12.2.126:37607)
0620 23:05:24.597477 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:24.597482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:24.597962 (+ 480us) server_negotiation.cc:407] Connection header received
0620 23:05:24.598131 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:24.598135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:24.598201 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:24.598296 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:24.599609 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.600306 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.601565 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.601970 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.602080 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:24.605604 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:24.605616 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:24.605619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:24.605650 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:24.608093 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.608646 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.608651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.608653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.608795 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.609109 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.609112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.609114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.609419 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:24.609592 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:24.609827 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 23:05:24.609984 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 23:05:24.610666 15369 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43210
I20260620 23:05:24.611904 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15430
Jun 20 23:05:24 dist-test-slave-7g8g krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996724, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:05:24.624838 15558 heartbeater.cc:503] Master 127.12.2.126:37607 was elected leader, sending a full tablet report...
I20260620 23:05:24.626094 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:24.613118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.12.2.65:45297)
0620 23:05:24.613668 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:24.613674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:24.613697 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:05:24.613745 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:24.613751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:24.613941 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:24.614056 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:24.615053 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.616028 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.617106 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.617647 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.617736 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:24.621025 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:24.621035 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:24.621048 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:24.621075 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:24.623618 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.624199 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.624206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.624208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.624313 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.624612 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.624616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.624619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.625018 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:24.625255 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:24.625590 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 23:05:24.625762 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":285,"threads_started":1}
W20260620 23:05:24.627233 15553 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58872
W20260620 23:05:24.628188 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15430: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:05:24.628364 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 15430
I20260620 23:05:24.640219 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15564
Jun 20 23:05:24 dist-test-slave-7g8g krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996724, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:05:24.653854 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:24.640980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.12.2.66:45131)
0620 23:05:24.641512 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:24.641519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:24.641543 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:05:24.641598 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:24.641603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:24.641804 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:24.641931 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:24.642990 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.643985 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.645120 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.645646 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.645735 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:24.649283 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:24.649293 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:24.649307 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:24.649337 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:24.651742 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.652295 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.652301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.652303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.652403 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.652655 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.652659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.652660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.652968 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:24.653173 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:24.653389 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 23:05:24.653574 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":265,"threads_started":1}
W20260620 23:05:24.655041 15687 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53596
W20260620 23:05:24.656019 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15564: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:05:24.656137 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 15564
I20260620 23:05:24.667596 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15697
Jun 20 23:05:24 dist-test-slave-7g8g krb5kdc[15324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996724, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:05:24.681913 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:24.668495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.12.2.67:35019)
0620 23:05:24.669103 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:24.669109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:24.669136 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:05:24.669188 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:24.669194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:24.669386 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:24.669503 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:24.670656 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.671675 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.672808 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:24.673316 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:24.673402 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:24.677123 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:24.677134 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:24.677148 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:24.677182 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:24.679681 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.680258 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.680264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.680266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.680376 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:24.680655 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:24.680658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:24.680660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:24.680964 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:24.681170 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:24.681364 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 23:05:24.681579 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":320,"threads_started":1}
W20260620 23:05:24.683197 15820 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60340
W20260620 23:05:24.684324 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15697: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:05:24.684487 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 15697
I20260620 23:05:24.696058 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15337
I20260620 23:05:24.697120 15399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:24.795527 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 15337
2026-06-20T23:05:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3305 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:24 dist-test-slave-7g8g krb5kdc[15842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:24 dist-test-slave-7g8g krb5kdc[15842](info): set up 2 sockets
Jun 20 23:05:24 dist-test-slave-7g8g krb5kdc[15842](info): commencing operation
krb5kdc: starting...
W20260620 23:05:25.787053 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:25 dist-test-slave-7g8g krb5kdc[15842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:05:25.956699 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:32951
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:32951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:26.278914 15855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:26.279281 15855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:26.279430 15855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:26.286808 15855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:26.286972 15855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:26.287065 15855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:26.287148 15855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:26.287216 15855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:26.297679 15855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:32951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:32951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:26.299162 15855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:26.300518 15855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:26.312220 15860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:26.312183 15863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:26.312250 15861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:26.313179 15855 server_base.cc:1034] running on GCE node
I20260620 23:05:26.313925 15855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:26.315297 15855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:26.316530 15855 hybrid_clock.cc:648] HybridClock initialized: now 1781996726316465 us; error 46 us; skew 500 ppm
I20260620 23:05:26.316876 15855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:26 dist-test-slave-7g8g krb5kdc[15842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996726, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:26.320039 15855 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:05:26.321460 15855 webserver.cc:460] Webserver started at http://127.12.2.126:42127/ using document root <none> and password file <none>
I20260620 23:05:26.322090 15855 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:26.322184 15855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:26.322430 15855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:26.325249 15855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "2501aad160b04c85bce5c30f449f8f5a"
format_stamp: "Formatted at 2026-06-20 23:05:26 on dist-test-slave-7g8g"
server_key: "63083dfefcee3ed1486ad0c8eb3ea0e8"
server_key_iv: "c9e0d2957b2bec9e4d0293d645c2be31"
server_key_version: "encryptionkey@0"
I20260620 23:05:26.325950 15855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "2501aad160b04c85bce5c30f449f8f5a"
format_stamp: "Formatted at 2026-06-20 23:05:26 on dist-test-slave-7g8g"
server_key: "63083dfefcee3ed1486ad0c8eb3ea0e8"
server_key_iv: "c9e0d2957b2bec9e4d0293d645c2be31"
server_key_version: "encryptionkey@0"
I20260620 23:05:26.330916 15855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260620 23:05:26.334256 15870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:26.335677 15855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:05:26.335830 15855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "2501aad160b04c85bce5c30f449f8f5a"
format_stamp: "Formatted at 2026-06-20 23:05:26 on dist-test-slave-7g8g"
server_key: "63083dfefcee3ed1486ad0c8eb3ea0e8"
server_key_iv: "c9e0d2957b2bec9e4d0293d645c2be31"
server_key_version: "encryptionkey@0"
I20260620 23:05:26.336004 15855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:26.349058 15855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:26.353000 15855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:26.353263 15855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:26.369812 15855 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:32951
I20260620 23:05:26.369815 15921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:32951 every 8 connection(s)
I20260620 23:05:26.371274 15855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:26.374902 15922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:26.381562 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 15855
I20260620 23:05:26.381759 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:26.382332 12297 external_mini_cluster.cc:1428] Setting key 492217d4d6c414fb6240fae2c1148ac2
I20260620 23:05:26.385073 15922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a: Bootstrap starting.
I20260620 23:05:26.388350 15922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:26.389562 15922 log.cc:824] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:26.392120 15922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a: No bootstrap required, opened a new log
Jun 20 23:05:26 dist-test-slave-7g8g krb5kdc[15842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:26.399259 15922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2501aad160b04c85bce5c30f449f8f5a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32951 } }
I20260620 23:05:26.399592 15922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:26.399675 15922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2501aad160b04c85bce5c30f449f8f5a, State: Initialized, Role: FOLLOWER
I20260620 23:05:26.400239 15922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2501aad160b04c85bce5c30f449f8f5a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32951 } }
I20260620 23:05:26.400460 15922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:26.400593 15922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:26.400751 15922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:26.403102 15922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2501aad160b04c85bce5c30f449f8f5a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32951 } }
I20260620 23:05:26.403200 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:26.384237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.12.2.126:32951)
0620 23:05:26.384946 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:26.384957 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:26.384997 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:05:26.386684 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:26.386706 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:26.387139 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:26.387441 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:26.389191 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:26.390508 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:26.391795 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:26.392468 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:26.392628 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:26.396184 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:26.396191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:26.396204 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:26.396240 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:26.399208 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:26.399910 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:26.399915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:26.399921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:26.400095 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:26.400433 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:26.400438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:26.400442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:26.401039 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:26.401331 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:26.401772 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 23:05:26.402178 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":317,"threads_started":1}
I20260620 23:05:26.403647 15922 leader_election.cc:302] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2501aad160b04c85bce5c30f449f8f5a; no voters:
I20260620 23:05:26.404260 15922 leader_election.cc:288] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:26.404340 15927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:26.405474 15927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [term 1 LEADER]: Becoming Leader. State: Replica: 2501aad160b04c85bce5c30f449f8f5a, State: Running, Role: LEADER
I20260620 23:05:26.405970 15927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2501aad160b04c85bce5c30f449f8f5a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32951 } }
I20260620 23:05:26.406333 15922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:26.408458 15929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2501aad160b04c85bce5c30f449f8f5a. Latest consensus state: current_term: 1 leader_uuid: "2501aad160b04c85bce5c30f449f8f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2501aad160b04c85bce5c30f449f8f5a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32951 } } }
I20260620 23:05:26.408895 15929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:26.412793 15928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2501aad160b04c85bce5c30f449f8f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2501aad160b04c85bce5c30f449f8f5a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32951 } } }
I20260620 23:05:26.413086 15928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:26.413318 15932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:26.420852 15932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:26.430140 15932 catalog_manager.cc:1347] Generated new cluster ID: 11edf6db3d1f4162848832f858ee83f7
I20260620 23:05:26.430258 15932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:26.479827 15932 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:26.481670 15932 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:26.499953 15932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2501aad160b04c85bce5c30f449f8f5a: Generated new TSK 0
I20260620 23:05:26.500846 15932 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:05:26.553956 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:32951
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:26.868610 15948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:26.868906 15948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:26.869026 15948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:26.876181 15948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:26.876309 15948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:26.876451 15948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:26.886030 15948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:32951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:26.887323 15948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:26.888592 15948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:26.901012 15953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:26.901048 15956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:26.901077 15954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:26.902027 15948 server_base.cc:1034] running on GCE node
I20260620 23:05:26.902813 15948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:26.903891 15948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:26.905141 15948 hybrid_clock.cc:648] HybridClock initialized: now 1781996726905083 us; error 49 us; skew 500 ppm
I20260620 23:05:26.905462 15948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:26 dist-test-slave-7g8g krb5kdc[15842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996726, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:26.908576 15948 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:05:26.910014 15948 webserver.cc:460] Webserver started at http://127.12.2.65:36663/ using document root <none> and password file <none>
I20260620 23:05:26.910712 15948 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:26.910825 15948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:26.911157 15948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:26.914235 15948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "954296bd2a9a44b9b37af78b30a98a83"
format_stamp: "Formatted at 2026-06-20 23:05:26 on dist-test-slave-7g8g"
server_key: "d33c268c5c2eed73dbd819b893d17d67"
server_key_iv: "c8cd6f0b0983beed69a4f6c4611ab9b6"
server_key_version: "encryptionkey@0"
I20260620 23:05:26.915047 15948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "954296bd2a9a44b9b37af78b30a98a83"
format_stamp: "Formatted at 2026-06-20 23:05:26 on dist-test-slave-7g8g"
server_key: "d33c268c5c2eed73dbd819b893d17d67"
server_key_iv: "c8cd6f0b0983beed69a4f6c4611ab9b6"
server_key_version: "encryptionkey@0"
I20260620 23:05:26.920140 15948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:05:26.923612 15963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:26.924889 15948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:05:26.925029 15948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "954296bd2a9a44b9b37af78b30a98a83"
format_stamp: "Formatted at 2026-06-20 23:05:26 on dist-test-slave-7g8g"
server_key: "d33c268c5c2eed73dbd819b893d17d67"
server_key_iv: "c8cd6f0b0983beed69a4f6c4611ab9b6"
server_key_version: "encryptionkey@0"
I20260620 23:05:26.925215 15948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:26.948882 15948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:26.952926 15948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:26.953181 15948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:26.954133 15948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:26.955474 15948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:26.955554 15948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:26.955670 15948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:26.955722 15948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:26.985148 15948 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:34827
I20260620 23:05:26.985217 16075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:34827 every 8 connection(s)
I20260620 23:05:26.986416 15948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:26.993142 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 15948
I20260620 23:05:26.993327 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:26.993927 12297 external_mini_cluster.cc:1428] Setting key f9160ca67604c759f1f23392b9fb574d
Jun 20 23:05:26 dist-test-slave-7g8g krb5kdc[15842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996726, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:27.005578 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:26.988926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:37957 (local address 127.12.2.126:32951)
0620 23:05:26.989337 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:26.989341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:26.990320 (+ 979us) server_negotiation.cc:407] Connection header received
0620 23:05:26.992202 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:26.992208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:26.992287 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:26.992379 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:26.994007 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:26.994842 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:26.996501 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:26.996895 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:26.997010 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:27.000293 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:27.000298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:27.000301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:27.000331 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:27.002714 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:27.003538 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:27.003542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:27.003544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:27.003665 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:27.004049 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:27.004052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:27.004054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:27.004302 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:27.004417 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:27.005159 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 23:05:27.005323 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":209,"threads_started":1}
I20260620 23:05:27.006990 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:26.989462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:32951 (local address 127.12.2.65:37957)
0620 23:05:26.990166 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:26.990206 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:26.991929 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:26.992701 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:26.992709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:26.993217 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:26.993836 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:26.993853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:26.995001 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:26.995005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:26.996367 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:26.996379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:26.997095 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:26.997102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:26.997328 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:26.997931 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:26.997953 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:27.000088 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:27.002846 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:27.002855 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:27.002872 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:27.003430 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:27.003748 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:27.003753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:27.003756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:27.003966 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:27.004408 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:27.004415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:27.004822 (+ 407us) client_negotiation.cc:769] Sending connection context
0620 23:05:27.005093 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 23:05:27.005605 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":279,"threads_started":1}
I20260620 23:05:27.008524 16076 heartbeater.cc:347] Connected to a master server at 127.12.2.126:32951
I20260620 23:05:27.008849 16076 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:27.009719 16076 heartbeater.cc:511] Master 127.12.2.126:32951 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
I20260620 23:05:27.012009 15887 ts_manager.cc:194] Registered new tserver with Master: 954296bd2a9a44b9b37af78b30a98a83 (127.12.2.65:34827)
I20260620 23:05:27.013404 15887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:37957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:05:27.034313 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:32951
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:27.341667 16082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:27.341913 16082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:27.342008 16082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:27.348982 16082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:27.349078 16082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:27.349193 16082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:27.358258 16082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:32951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:27.359476 16082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:27.360675 16082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:27.372730 16088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:27.373169 16087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:27.373399 16090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:27.374106 16082 server_base.cc:1034] running on GCE node
I20260620 23:05:27.374706 16082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:27.375699 16082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:27.376911 16082 hybrid_clock.cc:648] HybridClock initialized: now 1781996727376860 us; error 43 us; skew 500 ppm
I20260620 23:05:27.377223 16082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:27 dist-test-slave-7g8g krb5kdc[15842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996727, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:27.380419 16082 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:05:27.382128 16082 webserver.cc:460] Webserver started at http://127.12.2.66:35481/ using document root <none> and password file <none>
I20260620 23:05:27.382963 16082 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:27.383100 16082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:27.383414 16082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:27.387172 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "1f31287246524e3f8ccaffa6c42021cb"
format_stamp: "Formatted at 2026-06-20 23:05:27 on dist-test-slave-7g8g"
server_key: "c0ab694cb256b9c49c2fffc2f96ec502"
server_key_iv: "79a26f514fc181677be82bd1fd660201"
server_key_version: "encryptionkey@0"
I20260620 23:05:27.387934 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "1f31287246524e3f8ccaffa6c42021cb"
format_stamp: "Formatted at 2026-06-20 23:05:27 on dist-test-slave-7g8g"
server_key: "c0ab694cb256b9c49c2fffc2f96ec502"
server_key_iv: "79a26f514fc181677be82bd1fd660201"
server_key_version: "encryptionkey@0"
I20260620 23:05:27.392680 16082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:05:27.396054 16097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:27.397280 16082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:05:27.397404 16082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "1f31287246524e3f8ccaffa6c42021cb"
format_stamp: "Formatted at 2026-06-20 23:05:27 on dist-test-slave-7g8g"
server_key: "c0ab694cb256b9c49c2fffc2f96ec502"
server_key_iv: "79a26f514fc181677be82bd1fd660201"
server_key_version: "encryptionkey@0"
I20260620 23:05:27.397557 16082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:27.411917 16082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:27.415874 16082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:27.416103 16082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:27.416940 16082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:27.418154 16082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:27.418219 16082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:27.418322 16082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:27.418372 16082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:27.448377 16082 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:42509
I20260620 23:05:27.448398 16209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:42509 every 8 connection(s)
I20260620 23:05:27.450001 16082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:27.450439 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 16082
I20260620 23:05:27.450608 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:27.451143 12297 external_mini_cluster.cc:1428] Setting key ea814366987c93eeb605d5e8d344ef28
Jun 20 23:05:27 dist-test-slave-7g8g krb5kdc[15842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996727, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
I20260620 23:05:27.470268 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:27.452691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:44805 (local address 127.12.2.126:32951)
0620 23:05:27.452867 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:27.452874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:27.454182 (+ 1308us) server_negotiation.cc:407] Connection header received
0620 23:05:27.456102 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:27.456108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:27.456212 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:27.456299 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:27.457860 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.458633 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.460282 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.460849 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.460970 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:27.464328 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:27.464335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:27.464339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:27.464379 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:27.467529 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:27.468211 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:27.468215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:27.468217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:27.468336 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:27.468755 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:27.468758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:27.468760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:27.468991 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:27.469100 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:27.469775 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 23:05:27.469931 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:05:27.471297 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:27.453267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:32951 (local address 127.12.2.66:44805)
0620 23:05:27.454033 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:27.454076 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:27.455834 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:27.456566 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:27.456575 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:27.457085 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:27.457690 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:27.457708 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.458798 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.458802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:27.460126 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:27.460139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.461062 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.461067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:27.461277 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:27.461939 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:27.461963 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:27.464111 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:27.467659 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:27.467665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:27.467677 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:27.468094 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:27.468489 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:27.468493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:27.468496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:27.468670 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:27.469109 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:27.469115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:27.469399 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 23:05:27.469673 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 23:05:27.470032 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":319,"threads_started":1}
I20260620 23:05:27.472657 16210 heartbeater.cc:347] Connected to a master server at 127.12.2.126:32951
I20260620 23:05:27.472934 16210 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:27.473585 16210 heartbeater.cc:511] Master 127.12.2.126:32951 requested a full tablet report, sending...
I20260620 23:05:27.474835 15887 ts_manager.cc:194] Registered new tserver with Master: 1f31287246524e3f8ccaffa6c42021cb (127.12.2.66:42509)
I20260620 23:05:27.475623 15887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:44805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:05:27.490196 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:32951
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:27.795398 16215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:27.795697 16215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:27.795797 16215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:27.803364 16215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:27.803474 16215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:27.803599 16215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:27.813457 16215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:32951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:27.814658 16215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:27.815995 16215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:27.828210 16221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:27.828608 16220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:27.828634 16223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:27.829387 16215 server_base.cc:1034] running on GCE node
I20260620 23:05:27.829974 16215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:27.830889 16215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:27.832111 16215 hybrid_clock.cc:648] HybridClock initialized: now 1781996727832053 us; error 45 us; skew 500 ppm
I20260620 23:05:27.832401 16215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:27 dist-test-slave-7g8g krb5kdc[15842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996727, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:27.835335 16215 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:05:27.836684 16215 webserver.cc:460] Webserver started at http://127.12.2.67:35121/ using document root <none> and password file <none>
I20260620 23:05:27.837327 16215 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:27.837446 16215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:27.837721 16215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:27.840610 16215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "128c9dc90bb04fcf9643eb27c13820af"
format_stamp: "Formatted at 2026-06-20 23:05:27 on dist-test-slave-7g8g"
server_key: "33546d86c5724a6c67d652a3674ef575"
server_key_iv: "bfbdf3eda0e2ece4fe22e4858a3f829c"
server_key_version: "encryptionkey@0"
I20260620 23:05:27.841303 16215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "128c9dc90bb04fcf9643eb27c13820af"
format_stamp: "Formatted at 2026-06-20 23:05:27 on dist-test-slave-7g8g"
server_key: "33546d86c5724a6c67d652a3674ef575"
server_key_iv: "bfbdf3eda0e2ece4fe22e4858a3f829c"
server_key_version: "encryptionkey@0"
I20260620 23:05:27.845985 16215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 23:05:27.849305 16230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:27.850591 16215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:27.850734 16215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "128c9dc90bb04fcf9643eb27c13820af"
format_stamp: "Formatted at 2026-06-20 23:05:27 on dist-test-slave-7g8g"
server_key: "33546d86c5724a6c67d652a3674ef575"
server_key_iv: "bfbdf3eda0e2ece4fe22e4858a3f829c"
server_key_version: "encryptionkey@0"
I20260620 23:05:27.850916 16215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:27.875126 16215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:27.879622 16215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:27.879886 16215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:27.880807 16215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:27.882066 16215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:27.882139 16215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:27.882256 16215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:27.882309 16215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:27.910840 16215 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:42001
I20260620 23:05:27.910900 16342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:42001 every 8 connection(s)
I20260620 23:05:27.912109 16215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:27.916949 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 16215
I20260620 23:05:27.917109 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:27.917510 12297 external_mini_cluster.cc:1428] Setting key 197e47acef5860464dfc78894d64df5f
Jun 20 23:05:27 dist-test-slave-7g8g krb5kdc[15842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996727, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:27.930248 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:27.914436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:53661 (local address 127.12.2.126:32951)
0620 23:05:27.914628 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:27.914634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:27.915852 (+ 1218us) server_negotiation.cc:407] Connection header received
0620 23:05:27.917659 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:27.917664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:27.917756 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:27.917866 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:27.919543 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.920525 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.921962 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.922354 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.922444 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:27.925464 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:27.925469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:27.925472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:27.925502 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:27.927778 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:27.928340 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:27.928344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:27.928346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:27.928447 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:27.928848 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:27.928851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:27.928853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:27.929109 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:27.929219 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:27.929825 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 23:05:27.930007 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:05:27.931195 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:27.915028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:32951 (local address 127.12.2.67:53661)
0620 23:05:27.915700 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:27.915737 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:27.917403 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:27.918061 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:27.918071 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:27.918690 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:27.919398 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:27.919422 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.920654 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.920657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:27.921832 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:27.921842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.922507 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.922511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:27.922700 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:27.923281 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:27.923305 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:27.925285 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:27.927868 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:27.927874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:27.927884 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:27.928256 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:27.928594 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:27.928597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:27.928599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:27.928767 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:27.929200 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:27.929206 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:27.929474 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 23:05:27.929715 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 23:05:27.930062 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":252,"threads_started":1}
I20260620 23:05:27.932463 16343 heartbeater.cc:347] Connected to a master server at 127.12.2.126:32951
I20260620 23:05:27.932751 16343 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:27.933354 16343 heartbeater.cc:511] Master 127.12.2.126:32951 requested a full tablet report, sending...
I20260620 23:05:27.934568 15887 ts_manager.cc:194] Registered new tserver with Master: 128c9dc90bb04fcf9643eb27c13820af (127.12.2.67:42001)
I20260620 23:05:27.935364 15887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:53661
I20260620 23:05:27.944576 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:05:27 dist-test-slave-7g8g krb5kdc[15842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:05:27 dist-test-slave-7g8g krb5kdc[15842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:05:27.988168 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:27.974145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.12.2.65:34827)
0620 23:05:27.974695 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:27.974705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:27.974738 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:05:27.974809 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:27.974817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:27.975067 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:27.975212 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:27.976573 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.977565 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.978756 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.979354 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.979465 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:27.982924 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:27.982968 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:27.982987 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:27.983032 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:27.985562 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:27.986155 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:27.986162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:27.986164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:27.986283 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:27.986640 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:27.986645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:27.986648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:27.987013 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:27.987271 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:27.987558 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 23:05:27.987820 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":237,"threads_started":1}
Jun 20 23:05:27 dist-test-slave-7g8g krb5kdc[15842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:28.004354 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:27.991956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.12.2.126:32951)
0620 23:05:27.992169 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:27.992175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:27.992318 (+ 143us) server_negotiation.cc:407] Connection header received
0620 23:05:27.992440 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:27.992444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:27.992516 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:27.992592 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:27.993641 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.994377 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.995555 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:27.995932 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:27.996020 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:27.999570 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:27.999574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:27.999577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:27.999609 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:28.002055 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:28.002686 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:28.002695 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:28.002697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:28.002802 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:28.003189 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:28.003192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:28.003194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:28.003492 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:28.003689 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:28.003921 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 23:05:28.004071 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 23:05:28.004936 15887 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37722
I20260620 23:05:28.005867 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15948
I20260620 23:05:28.016239 16076 heartbeater.cc:503] Master 127.12.2.126:32951 was elected leader, sending a full tablet report...
I20260620 23:05:28.017650 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:28.006709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.12.2.65:34827)
0620 23:05:28.006910 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:28.006915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:28.007149 (+ 234us) server_negotiation.cc:407] Connection header received
0620 23:05:28.007304 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:28.007307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:28.007396 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:28.007473 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:28.008696 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:28.009350 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:28.010572 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:28.010978 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:28.011069 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:28.013109 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:28.013113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:28.013116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:28.013138 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:28.015445 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:28.016052 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:28.016055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:28.016057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:28.016159 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:28.016522 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:28.016526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:28.016528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:28.016777 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:28.016984 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:28.017268 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 23:05:28.017420 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 23:05:28.018348 16071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:28.098734 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 15948
I20260620 23:05:28.111804 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 16082
Jun 20 23:05:28 dist-test-slave-7g8g krb5kdc[15842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:05:28.126555 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:28.112657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.12.2.66:42509)
0620 23:05:28.113136 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:28.113143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:28.113167 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:05:28.113224 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:28.113231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:28.113439 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:28.113581 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:28.114725 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:28.115737 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:28.116977 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:28.117510 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:28.117630 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:28.121390 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:28.121400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:28.121415 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:28.121444 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:28.123968 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:28.124556 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:28.124563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:28.124565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:28.124687 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:28.125019 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:28.125024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:28.125026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:28.125357 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:28.125652 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:28.125945 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 23:05:28.126182 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":244,"threads_started":1}
I20260620 23:05:28.128160 16205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:28.208534 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 16082
I20260620 23:05:28.221118 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 16215
Jun 20 23:05:28 dist-test-slave-7g8g krb5kdc[15842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:05:28.236270 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:28.222122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48092 (local address 127.12.2.67:42001)
0620 23:05:28.222693 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:28.222699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:28.222726 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:05:28.222784 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:28.222790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:28.223027 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:28.223182 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:28.224379 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:28.225377 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:28.226591 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:28.227217 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:28.227333 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:28.231296 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:28.231306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:28.231319 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:28.231354 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:28.233720 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:28.234314 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:28.234323 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:28.234325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:28.234443 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:28.234771 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:28.234776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:28.234779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:28.235148 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:28.235436 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:28.235705 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 23:05:28.235945 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":297,"threads_started":1}
I20260620 23:05:28.237469 16338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:28.324237 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 16215
I20260620 23:05:28.336036 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 15855
I20260620 23:05:28.337221 15917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:28.434899 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 15855
2026-06-20T23:05:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3639 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:28 dist-test-slave-7g8g krb5kdc[16362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:28 dist-test-slave-7g8g krb5kdc[16362](info): set up 2 sockets
Jun 20 23:05:28 dist-test-slave-7g8g krb5kdc[16362](info): commencing operation
krb5kdc: starting...
W20260620 23:05:29.430895 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:29 dist-test-slave-7g8g krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:05:29.604111 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:45805
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:45805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:29.898880 16375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:29.899161 16375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:29.899291 16375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:29.906152 16375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:29.906244 16375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:29.906311 16375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:29.906385 16375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:29.906450 16375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:29.915756 16375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:45805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:45805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:29.916944 16375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:29.918174 16375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:29.928372 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:29.928427 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:29.928395 16383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:29.929188 16375 server_base.cc:1034] running on GCE node
I20260620 23:05:29.929874 16375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:29.931169 16375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:29.932351 16375 hybrid_clock.cc:648] HybridClock initialized: now 1781996729932309 us; error 32 us; skew 500 ppm
I20260620 23:05:29.932674 16375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:29 dist-test-slave-7g8g krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996729, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:29.939493 16375 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:05:29.940563 16375 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:29.940657 16375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:29.940943 16375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:29.943737 16375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "8149d439bf4c43839551f014e634a59f"
format_stamp: "Formatted at 2026-06-20 23:05:29 on dist-test-slave-7g8g"
server_key: "49b53fe60054f8fe0e901b1c9d6e2aee"
server_key_iv: "1132e3c15e40fc8fea871f0891acfb2e"
server_key_version: "encryptionkey@0"
I20260620 23:05:29.944429 16375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "8149d439bf4c43839551f014e634a59f"
format_stamp: "Formatted at 2026-06-20 23:05:29 on dist-test-slave-7g8g"
server_key: "49b53fe60054f8fe0e901b1c9d6e2aee"
server_key_iv: "1132e3c15e40fc8fea871f0891acfb2e"
server_key_version: "encryptionkey@0"
I20260620 23:05:29.949136 16375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 23:05:29.952504 16389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:29.953795 16375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:05:29.954003 16375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "8149d439bf4c43839551f014e634a59f"
format_stamp: "Formatted at 2026-06-20 23:05:29 on dist-test-slave-7g8g"
server_key: "49b53fe60054f8fe0e901b1c9d6e2aee"
server_key_iv: "1132e3c15e40fc8fea871f0891acfb2e"
server_key_version: "encryptionkey@0"
I20260620 23:05:29.954221 16375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:29.977732 16375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:29.981576 16375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:29.981789 16375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:29.997452 16375 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:45805
I20260620 23:05:29.997432 16440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:45805 every 8 connection(s)
I20260620 23:05:29.998397 16375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:29.998697 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 16375
I20260620 23:05:29.998893 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:29.999702 12297 external_mini_cluster.cc:1428] Setting key 639f15cc2a7ed2d424ba3136b74400c4
I20260620 23:05:30.002724 16441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:05:30 dist-test-slave-7g8g krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:30.015024 16441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f: Bootstrap starting.
I20260620 23:05:30.018049 16441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:30.019019 16441 log.cc:824] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:30.022140 16441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f: No bootstrap required, opened a new log
I20260620 23:05:30.022753 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.001730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.12.2.126:45805)
0620 23:05:30.002485 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.002497 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.002540 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:05:30.004213 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.004236 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.004618 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.004959 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:30.006694 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.008165 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.009906 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.010641 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.010788 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.015298 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:30.015308 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:30.015324 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:30.015368 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:30.018488 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.019199 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.019207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.019214 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.019417 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.019871 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.019875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.019877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.020479 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:30.020750 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.021100 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.021499 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":339,"threads_started":1}
I20260620 23:05:30.029642 16441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149d439bf4c43839551f014e634a59f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45805 } }
I20260620 23:05:30.029955 16441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:30.030025 16441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8149d439bf4c43839551f014e634a59f, State: Initialized, Role: FOLLOWER
I20260620 23:05:30.030463 16441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149d439bf4c43839551f014e634a59f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45805 } }
I20260620 23:05:30.030660 16441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:30.030747 16441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:30.030855 16441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:30.032816 16441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149d439bf4c43839551f014e634a59f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45805 } }
I20260620 23:05:30.033228 16441 leader_election.cc:302] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8149d439bf4c43839551f014e634a59f; no voters:
I20260620 23:05:30.033735 16441 leader_election.cc:288] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:30.033846 16446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:30.034411 16446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [term 1 LEADER]: Becoming Leader. State: Replica: 8149d439bf4c43839551f014e634a59f, State: Running, Role: LEADER
I20260620 23:05:30.035033 16446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149d439bf4c43839551f014e634a59f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45805 } }
I20260620 23:05:30.035630 16441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:30.037355 16447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8149d439bf4c43839551f014e634a59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149d439bf4c43839551f014e634a59f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45805 } } }
I20260620 23:05:30.037614 16447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:30.038992 16451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:30.039244 16448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8149d439bf4c43839551f014e634a59f. Latest consensus state: current_term: 1 leader_uuid: "8149d439bf4c43839551f014e634a59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8149d439bf4c43839551f014e634a59f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45805 } } }
I20260620 23:05:30.039512 16448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:30.042975 16451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:30.051414 16451 catalog_manager.cc:1347] Generated new cluster ID: 0a1f4b665be149ac91c89e9d7c89ade7
I20260620 23:05:30.051538 16451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:30.084420 16451 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:30.085668 16451 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:30.095361 16451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f: Generated new TSK 0
I20260620 23:05:30.096181 16451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:05:30.140667 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:45805
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:30.446741 16467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:30.447031 16467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:30.447181 16467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:30.454193 16467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:30.454283 16467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:30.454434 16467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:30.463822 16467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:45805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:30.465049 16467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:30.466317 16467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:30.478057 16473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.478121 16472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.478142 16475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:30.479171 16467 server_base.cc:1034] running on GCE node
I20260620 23:05:30.479728 16467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:30.480559 16467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:30.481739 16467 hybrid_clock.cc:648] HybridClock initialized: now 1781996730481700 us; error 36 us; skew 500 ppm
I20260620 23:05:30.482013 16467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:30 dist-test-slave-7g8g krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996730, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:30.484933 16467 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:05:30.485962 16467 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:30.486047 16467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:30.486299 16467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:30.489043 16467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "d194bbcb5bf64bfebb75970d19d70c4e"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-7g8g"
server_key: "2d58f8cd9c851eaa382becc06976a395"
server_key_iv: "5b9a74c9a33327dc9b2d269356831562"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.489753 16467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "d194bbcb5bf64bfebb75970d19d70c4e"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-7g8g"
server_key: "2d58f8cd9c851eaa382becc06976a395"
server_key_iv: "5b9a74c9a33327dc9b2d269356831562"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.494439 16467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:05:30.497676 16481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.498996 16467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:30.499143 16467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "d194bbcb5bf64bfebb75970d19d70c4e"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-7g8g"
server_key: "2d58f8cd9c851eaa382becc06976a395"
server_key_iv: "5b9a74c9a33327dc9b2d269356831562"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.499326 16467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:30.513070 16467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:30.516692 16467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:30.516913 16467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:30.517789 16467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:30.519048 16467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:30.519145 16467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.519244 16467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:30.519312 16467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.548308 16467 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:38641
I20260620 23:05:30.548346 16593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:38641 every 8 connection(s)
I20260620 23:05:30.549084 16467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:30.558642 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 16467
I20260620 23:05:30.558852 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:30.559607 12297 external_mini_cluster.cc:1428] Setting key 0772d2e7b6af34801201c6ea435c89bf
Jun 20 23:05:30 dist-test-slave-7g8g krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996730, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:30.568792 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.551287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:39359 (local address 127.12.2.126:45805)
0620 23:05:30.551711 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.551717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.552638 (+ 921us) server_negotiation.cc:407] Connection header received
0620 23:05:30.554436 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.554440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.554502 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.554592 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:30.555977 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.556684 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.558160 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.558608 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.558725 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.563755 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:30.563759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:30.563763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:30.563792 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:30.566051 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.566691 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.566696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.566700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.566807 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.567205 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.567209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.567212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.567452 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:30.567584 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.568213 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.568442 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":201,"threads_started":1}
I20260620 23:05:30.569645 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.551853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:45805 (local address 127.12.2.65:39359)
0620 23:05:30.552491 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.552532 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:30.554205 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:30.554775 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:30.554782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:30.555308 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:30.555838 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.555854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.556784 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.556787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.558029 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.558039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.558817 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.558824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.559124 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.559780 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:30.559813 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:30.563546 (+ 3733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:30.566173 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:30.566182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:30.566199 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:30.566587 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:30.566903 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:30.566907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:30.566911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:30.567109 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:30.567564 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:30.567571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:30.567873 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 23:05:30.568103 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 23:05:30.568577 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":253,"threads_started":1}
I20260620 23:05:30.571045 16594 heartbeater.cc:347] Connected to a master server at 127.12.2.126:45805
I20260620 23:05:30.571339 16594 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:30.572088 16594 heartbeater.cc:511] Master 127.12.2.126:45805 requested a full tablet report, sending...
I20260620 23:05:30.574412 16406 ts_manager.cc:194] Registered new tserver with Master: d194bbcb5bf64bfebb75970d19d70c4e (127.12.2.65:38641)
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
I20260620 23:05:30.576413 16406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:39359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:05:30.598826 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:45805
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:30.895798 16600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:30.896071 16600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:30.896173 16600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:30.903301 16600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:30.903401 16600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:30.903517 16600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:30.912873 16600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:45805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:30.914077 16600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:30.915314 16600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:30.926990 16605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.926954 16608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.926965 16606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:30.927795 16600 server_base.cc:1034] running on GCE node
I20260620 23:05:30.928299 16600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:30.929078 16600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:30.930265 16600 hybrid_clock.cc:648] HybridClock initialized: now 1781996730930217 us; error 40 us; skew 500 ppm
I20260620 23:05:30.930550 16600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:30 dist-test-slave-7g8g krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996730, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:30.933357 16600 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:05:30.934350 16600 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:30.934442 16600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:30.934725 16600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:30.937692 16600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "21411c22b4744644a18bd93cf99476a4"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-7g8g"
server_key: "d0698bd43ae37039c248b4024a9f4aa4"
server_key_iv: "c9c01a0399f5855a001a26082ba8a985"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.938436 16600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "21411c22b4744644a18bd93cf99476a4"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-7g8g"
server_key: "d0698bd43ae37039c248b4024a9f4aa4"
server_key_iv: "c9c01a0399f5855a001a26082ba8a985"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.943192 16600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260620 23:05:30.946369 16614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.947669 16600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:30.947810 16600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "21411c22b4744644a18bd93cf99476a4"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-7g8g"
server_key: "d0698bd43ae37039c248b4024a9f4aa4"
server_key_iv: "c9c01a0399f5855a001a26082ba8a985"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.948009 16600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:30.962180 16600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:30.966048 16600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:30.966287 16600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:30.967177 16600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:30.968387 16600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:30.968477 16600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.968585 16600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:30.968653 16600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.996480 16600 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:34699
I20260620 23:05:30.996531 16726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:34699 every 8 connection(s)
I20260620 23:05:30.997385 16600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:31.004240 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 16600
I20260620 23:05:31.004422 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:31.004845 12297 external_mini_cluster.cc:1428] Setting key fa43a1fe10c95a13e8629e2860b5608e
Jun 20 23:05:31 dist-test-slave-7g8g krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996730, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:31.017732 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.999774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:49189 (local address 127.12.2.126:45805)
0620 23:05:30.999968 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.999974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.001206 (+ 1232us) server_negotiation.cc:407] Connection header received
0620 23:05:31.003697 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.003702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.003794 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.003885 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:31.005746 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.006584 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.008139 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.008561 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.008666 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.011861 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:31.011865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:31.011868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:31.011905 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:31.014802 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:31.015719 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:31.015723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:31.015725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:31.015851 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:31.016247 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:31.016249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:31.016251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:31.016489 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:31.016591 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.017338 (+ 747us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.017496 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:05:31.019062 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.000263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:45805 (local address 127.12.2.66:49189)
0620 23:05:31.001020 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.001069 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.003379 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.004125 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.004136 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.004842 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:31.005573 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.005595 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.006711 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.006715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.008010 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.008022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.008772 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.008780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.008996 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.009602 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:31.009628 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:31.011671 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:31.014965 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:31.014974 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:31.014991 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:31.015579 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:31.015943 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:31.015948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:31.015951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:31.016159 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:31.016588 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:31.016595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:31.016986 (+ 391us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.017274 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.017768 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":296,"threads_started":1}
I20260620 23:05:31.020437 16727 heartbeater.cc:347] Connected to a master server at 127.12.2.126:45805
I20260620 23:05:31.020725 16727 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:31.021373 16727 heartbeater.cc:511] Master 127.12.2.126:45805 requested a full tablet report, sending...
I20260620 23:05:31.022708 16406 ts_manager.cc:194] Registered new tserver with Master: 21411c22b4744644a18bd93cf99476a4 (127.12.2.66:34699)
I20260620 23:05:31.023515 16406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:49189
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:05:31.046108 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:45805
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:36625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:31.346029 16732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:31.346292 16732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:31.346383 16732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:31.353376 16732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:31.353492 16732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:31.353610 16732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:31.363070 16732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:36625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:45805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:31.364322 16732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:31.365582 16732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:31.377578 16737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:31.377647 16738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:31.377673 16740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:31.378705 16732 server_base.cc:1034] running on GCE node
I20260620 23:05:31.379314 16732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:31.380162 16732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:31.381343 16732 hybrid_clock.cc:648] HybridClock initialized: now 1781996731381297 us; error 42 us; skew 500 ppm
I20260620 23:05:31.381650 16732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:31 dist-test-slave-7g8g krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996731, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:31.384647 16732 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:05:31.385641 16732 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:31.385722 16732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:31.385964 16732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:31.388705 16732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "87a9918160cf4cd1bdfb14cad66445a6"
format_stamp: "Formatted at 2026-06-20 23:05:31 on dist-test-slave-7g8g"
server_key: "980b02420a32329002b58bfec3014e36"
server_key_iv: "f0f6404a09896f67bc6a0e08001a161e"
server_key_version: "encryptionkey@0"
I20260620 23:05:31.389395 16732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "87a9918160cf4cd1bdfb14cad66445a6"
format_stamp: "Formatted at 2026-06-20 23:05:31 on dist-test-slave-7g8g"
server_key: "980b02420a32329002b58bfec3014e36"
server_key_iv: "f0f6404a09896f67bc6a0e08001a161e"
server_key_version: "encryptionkey@0"
I20260620 23:05:31.394026 16732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 23:05:31.397219 16746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:31.398420 16732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:05:31.398582 16732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "87a9918160cf4cd1bdfb14cad66445a6"
format_stamp: "Formatted at 2026-06-20 23:05:31 on dist-test-slave-7g8g"
server_key: "980b02420a32329002b58bfec3014e36"
server_key_iv: "f0f6404a09896f67bc6a0e08001a161e"
server_key_version: "encryptionkey@0"
I20260620 23:05:31.398765 16732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:31.418074 16732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:31.421993 16732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:31.422228 16732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:31.423132 16732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:31.424368 16732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:31.424458 16732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:31.424575 16732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:31.424641 16732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:31.456557 16732 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:32965
I20260620 23:05:31.456595 16858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:32965 every 8 connection(s)
I20260620 23:05:31.457384 16732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:31.461973 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 16732
I20260620 23:05:31.462128 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:31.462517 12297 external_mini_cluster.cc:1428] Setting key b2212868201818ba289fa1d4e92b641c
Jun 20 23:05:31 dist-test-slave-7g8g krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996731, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:31.475294 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.459741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:43303 (local address 127.12.2.126:45805)
0620 23:05:31.459902 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.459906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.461029 (+ 1123us) server_negotiation.cc:407] Connection header received
0620 23:05:31.462816 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.462822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.462961 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.463056 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:31.464803 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.465683 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.467169 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.467579 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.467680 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.470652 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:31.470656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:31.470660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:31.470695 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:31.472852 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:31.473433 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:31.473436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:31.473438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:31.473538 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:31.473834 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:31.473837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:31.473839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:31.474077 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:31.474175 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.474865 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.475054 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:05:31.476325 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.460226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:45805 (local address 127.12.2.67:43303)
0620 23:05:31.460877 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.460912 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.462546 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.463302 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.463310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.463936 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:31.464650 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.464674 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.465813 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.465817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.467064 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.467075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.467750 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.467755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.467929 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.468508 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:31.468537 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:31.470486 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:31.472958 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:31.472964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:31.472975 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:31.473353 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:31.473608 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:31.473611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:31.473613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:31.473763 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:31.474190 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:31.474199 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:31.474592 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.474810 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.475206 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":266,"threads_started":1}
I20260620 23:05:31.477659 16859 heartbeater.cc:347] Connected to a master server at 127.12.2.126:45805
I20260620 23:05:31.477924 16859 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:31.478497 16859 heartbeater.cc:511] Master 127.12.2.126:45805 requested a full tablet report, sending...
I20260620 23:05:31.479779 16405 ts_manager.cc:194] Registered new tserver with Master: 87a9918160cf4cd1bdfb14cad66445a6 (127.12.2.67:32965)
I20260620 23:05:31.480561 16405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:43303
I20260620 23:05:31.489053 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:05:31.505059 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.493302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.12.2.126:45805)
0620 23:05:31.493470 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.493477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.493915 (+ 438us) server_negotiation.cc:407] Connection header received
0620 23:05:31.494032 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.494036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.494106 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.494192 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:31.495572 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.496329 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.497480 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.497901 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.498017 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.500049 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:31.500054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:31.500057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:31.500087 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:31.502584 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:31.503263 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:31.503267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:31.503269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:31.503412 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:31.503751 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:31.503757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:31.503759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:31.504053 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:31.504262 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.504576 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.504772 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:05:31.508657 16405 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:31.515219 16405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:31.539294 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.526325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:32965 (local address 127.0.0.1:42300)
0620 23:05:31.527480 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.527522 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.527692 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.530124 (+ 2432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.530130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.530158 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:31.530742 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.530754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.533118 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.533125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.536374 (+ 3249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.536389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.538277 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.538283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.538460 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.538512 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.538639 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.538833 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":25,"thread_start_us":2061,"threads_started":1}
I20260620 23:05:31.539875 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.526493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.12.2.67:32965)
0620 23:05:31.529489 (+ 2996us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.529496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.529529 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:05:31.529637 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.529646 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.529890 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.530980 (+ 1090us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:31.531266 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.532948 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.536598 (+ 3650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.538101 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.538229 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.538452 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.539408 (+ 956us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.539561 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2822,"thread_start_us":2689,"threads_started":1}
I20260620 23:05:31.539902 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.527198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.12.2.65:38641)
0620 23:05:31.528250 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.528261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.529430 (+ 1169us) server_negotiation.cc:407] Connection header received
0620 23:05:31.529537 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.529544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.529819 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.530878 (+ 1059us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:31.532269 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.533849 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.537171 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.538552 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.538663 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.538843 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.539112 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.539283 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":775,"threads_started":1}
I20260620 23:05:31.540455 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.526619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38641 (local address 127.0.0.1:41250)
0620 23:05:31.529263 (+ 2644us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.529281 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.529395 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.530060 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.530067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.530099 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:31.530667 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.530699 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.534015 (+ 3316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.534021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.537003 (+ 2982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.537017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.538748 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.538754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.538913 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.538975 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.539061 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.539195 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2465,"mutex_wait_us":1,"thread_start_us":1994,"threads_started":1}
I20260620 23:05:31.543704 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.529288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:34699 (local address 127.0.0.1:33618)
0620 23:05:31.536796 (+ 7508us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.536815 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.536936 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.537559 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.537563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.537591 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:31.537908 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.537916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.539817 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.539822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.541730 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.541741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.542969 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.542976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.543178 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.543201 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.543276 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.543388 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7385,"thread_start_us":7247,"threads_started":1}
I20260620 23:05:31.543747 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.529698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.12.2.66:34699)
0620 23:05:31.535821 (+ 6123us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.535830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.536921 (+ 1091us) server_negotiation.cc:407] Connection header received
0620 23:05:31.537092 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.537101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.537352 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.537568 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:31.538036 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.539452 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.541871 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.542769 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.542881 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.543108 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.543328 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.543483 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5874,"thread_start_us":5761,"threads_started":1}
I20260620 23:05:31.544328 16529 tablet_service.cc:1467] Processing CreateTablet for tablet 2e901d068ada4ad184326ccfd9dce83d (DEFAULT_TABLE table=test-table [id=718909b6b52c487b898e75e1d712559a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:31.544329 16794 tablet_service.cc:1467] Processing CreateTablet for tablet 2e901d068ada4ad184326ccfd9dce83d (DEFAULT_TABLE table=test-table [id=718909b6b52c487b898e75e1d712559a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:31.545778 16794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e901d068ada4ad184326ccfd9dce83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:31.545782 16529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e901d068ada4ad184326ccfd9dce83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:31.546994 16662 tablet_service.cc:1467] Processing CreateTablet for tablet 2e901d068ada4ad184326ccfd9dce83d (DEFAULT_TABLE table=test-table [id=718909b6b52c487b898e75e1d712559a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:31.548182 16662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e901d068ada4ad184326ccfd9dce83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:31.559737 16877 tablet_bootstrap.cc:492] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: Bootstrap starting.
I20260620 23:05:31.562502 16878 tablet_bootstrap.cc:492] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: Bootstrap starting.
I20260620 23:05:31.563346 16877 tablet_bootstrap.cc:654] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:31.563828 16879 tablet_bootstrap.cc:492] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: Bootstrap starting.
I20260620 23:05:31.565094 16877 log.cc:824] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:31.565678 16878 tablet_bootstrap.cc:654] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:31.566839 16878 log.cc:824] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:31.566990 16879 tablet_bootstrap.cc:654] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:31.568120 16879 log.cc:824] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:31.568341 16877 tablet_bootstrap.cc:492] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: No bootstrap required, opened a new log
I20260620 23:05:31.568609 16877 ts_tablet_manager.cc:1397] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260620 23:05:31.569373 16878 tablet_bootstrap.cc:492] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: No bootstrap required, opened a new log
I20260620 23:05:31.569599 16878 ts_tablet_manager.cc:1397] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 23:05:31.576923 16878 raft_consensus.cc:348] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a9918160cf4cd1bdfb14cad66445a6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32965 } } peers { permanent_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38641 } } peers { permanent_uuid: "21411c22b4744644a18bd93cf99476a4" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 34699 } }
I20260620 23:05:31.577296 16878 raft_consensus.cc:374] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:31.577383 16878 raft_consensus.cc:729] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21411c22b4744644a18bd93cf99476a4, State: Initialized, Role: FOLLOWER
I20260620 23:05:31.577651 16879 tablet_bootstrap.cc:492] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: No bootstrap required, opened a new log
I20260620 23:05:31.577911 16879 ts_tablet_manager.cc:1397] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260620 23:05:31.577942 16878 consensus_queue.cc:260] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a9918160cf4cd1bdfb14cad66445a6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32965 } } peers { permanent_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38641 } } peers { permanent_uuid: "21411c22b4744644a18bd93cf99476a4" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 34699 } }
I20260620 23:05:31.579028 16727 heartbeater.cc:503] Master 127.12.2.126:45805 was elected leader, sending a full tablet report...
I20260620 23:05:31.578723 16877 raft_consensus.cc:348] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a9918160cf4cd1bdfb14cad66445a6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32965 } } peers { permanent_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38641 } } peers { permanent_uuid: "21411c22b4744644a18bd93cf99476a4" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 34699 } }
I20260620 23:05:31.579164 16877 raft_consensus.cc:374] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:31.579257 16877 raft_consensus.cc:729] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87a9918160cf4cd1bdfb14cad66445a6, State: Initialized, Role: FOLLOWER
I20260620 23:05:31.579784 16878 ts_tablet_manager.cc:1428] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260620 23:05:31.579814 16877 consensus_queue.cc:260] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a9918160cf4cd1bdfb14cad66445a6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32965 } } peers { permanent_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38641 } } peers { permanent_uuid: "21411c22b4744644a18bd93cf99476a4" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 34699 } }
I20260620 23:05:31.580421 16594 heartbeater.cc:503] Master 127.12.2.126:45805 was elected leader, sending a full tablet report...
I20260620 23:05:31.585902 16859 heartbeater.cc:503] Master 127.12.2.126:45805 was elected leader, sending a full tablet report...
I20260620 23:05:31.586625 16877 ts_tablet_manager.cc:1428] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260620 23:05:31.588020 16879 raft_consensus.cc:348] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a9918160cf4cd1bdfb14cad66445a6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32965 } } peers { permanent_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38641 } } peers { permanent_uuid: "21411c22b4744644a18bd93cf99476a4" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 34699 } }
I20260620 23:05:31.588322 16879 raft_consensus.cc:374] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:31.588413 16879 raft_consensus.cc:729] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d194bbcb5bf64bfebb75970d19d70c4e, State: Initialized, Role: FOLLOWER
I20260620 23:05:31.588949 16879 consensus_queue.cc:260] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a9918160cf4cd1bdfb14cad66445a6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32965 } } peers { permanent_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38641 } } peers { permanent_uuid: "21411c22b4744644a18bd93cf99476a4" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 34699 } }
I20260620 23:05:31.590245 16879 ts_tablet_manager.cc:1428] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
W20260620 23:05:31.708814 16860 tablet.cc:2369] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:05:31.749195 16728 tablet.cc:2369] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:31.772405 16885 raft_consensus.cc:482] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:31.772706 16885 raft_consensus.cc:504] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a9918160cf4cd1bdfb14cad66445a6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32965 } } peers { permanent_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38641 } } peers { permanent_uuid: "21411c22b4744644a18bd93cf99476a4" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 34699 } }
I20260620 23:05:31.774359 16885 leader_election.cc:288] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87a9918160cf4cd1bdfb14cad66445a6 (127.12.2.67:32965), 21411c22b4744644a18bd93cf99476a4 (127.12.2.66:34699)
I20260620 23:05:31.780915 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.774707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:38915 (local address 127.12.2.67:32965)
0620 23:05:31.774873 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.774877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.775290 (+ 413us) server_negotiation.cc:407] Connection header received
0620 23:05:31.775498 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.775504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.775579 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.775678 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:31.776333 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.777248 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.779108 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.780037 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.780128 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.780200 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.780563 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.780679 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:05:31.780936 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.774599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:32965 (local address 127.12.2.65:38915)
0620 23:05:31.775202 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.775220 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.775352 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.775810 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.775814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.775850 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:31.776192 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.776204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.777370 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.777375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.778968 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.778981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.780239 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.780245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.780412 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.780444 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.780543 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.780670 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":285,"threads_started":1}
I20260620 23:05:31.781342 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.775464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:44275 (local address 127.12.2.66:34699)
0620 23:05:31.775719 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.775726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.776086 (+ 360us) server_negotiation.cc:407] Connection header received
0620 23:05:31.776218 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.776222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.776293 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.776368 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:31.776917 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.777778 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.779582 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.780459 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.780566 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.780648 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.781008 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.781122 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:05:31.781374 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.775351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:34699 (local address 127.12.2.65:44275)
0620 23:05:31.775986 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.776006 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.776104 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.776392 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.776397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.776420 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:31.776792 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.776803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.777910 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.777913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.779434 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.779448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.780664 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.780670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.780838 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.780867 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.780958 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.781070 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":356,"threads_started":1}
I20260620 23:05:31.782124 16682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e901d068ada4ad184326ccfd9dce83d" candidate_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21411c22b4744644a18bd93cf99476a4" is_pre_election: true
I20260620 23:05:31.782146 16814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e901d068ada4ad184326ccfd9dce83d" candidate_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87a9918160cf4cd1bdfb14cad66445a6" is_pre_election: true
I20260620 23:05:31.782543 16682 raft_consensus.cc:2454] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d194bbcb5bf64bfebb75970d19d70c4e in term 0.
I20260620 23:05:31.782543 16814 raft_consensus.cc:2454] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d194bbcb5bf64bfebb75970d19d70c4e in term 0.
I20260620 23:05:31.783108 16483 leader_election.cc:302] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87a9918160cf4cd1bdfb14cad66445a6, d194bbcb5bf64bfebb75970d19d70c4e; no voters:
I20260620 23:05:31.783427 16885 raft_consensus.cc:2781] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:31.783586 16885 raft_consensus.cc:482] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:31.783684 16885 raft_consensus.cc:3037] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:31.785799 16885 raft_consensus.cc:504] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a9918160cf4cd1bdfb14cad66445a6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32965 } } peers { permanent_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38641 } } peers { permanent_uuid: "21411c22b4744644a18bd93cf99476a4" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 34699 } }
I20260620 23:05:31.786305 16885 leader_election.cc:288] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [CANDIDATE]: Term 1 election: Requested vote from peers 87a9918160cf4cd1bdfb14cad66445a6 (127.12.2.67:32965), 21411c22b4744644a18bd93cf99476a4 (127.12.2.66:34699)
I20260620 23:05:31.786732 16814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e901d068ada4ad184326ccfd9dce83d" candidate_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87a9918160cf4cd1bdfb14cad66445a6"
I20260620 23:05:31.786788 16682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e901d068ada4ad184326ccfd9dce83d" candidate_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21411c22b4744644a18bd93cf99476a4"
I20260620 23:05:31.786902 16814 raft_consensus.cc:3037] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:31.786988 16682 raft_consensus.cc:3037] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:31.788981 16814 raft_consensus.cc:2454] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d194bbcb5bf64bfebb75970d19d70c4e in term 1.
I20260620 23:05:31.788982 16682 raft_consensus.cc:2454] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d194bbcb5bf64bfebb75970d19d70c4e in term 1.
I20260620 23:05:31.789319 16483 leader_election.cc:302] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87a9918160cf4cd1bdfb14cad66445a6, d194bbcb5bf64bfebb75970d19d70c4e; no voters:
I20260620 23:05:31.789592 16885 raft_consensus.cc:2781] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:31.789964 16885 raft_consensus.cc:686] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 1 LEADER]: Becoming Leader. State: Replica: d194bbcb5bf64bfebb75970d19d70c4e, State: Running, Role: LEADER
I20260620 23:05:31.790427 16885 consensus_queue.cc:237] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a9918160cf4cd1bdfb14cad66445a6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32965 } } peers { permanent_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38641 } } peers { permanent_uuid: "21411c22b4744644a18bd93cf99476a4" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 34699 } }
I20260620 23:05:31.794836 16406 catalog_manager.cc:5495] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e reported cstate change: term changed from 0 to 1, leader changed from <none> to d194bbcb5bf64bfebb75970d19d70c4e (127.12.2.65). New cstate: current_term: 1 leader_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87a9918160cf4cd1bdfb14cad66445a6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d194bbcb5bf64bfebb75970d19d70c4e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21411c22b4744644a18bd93cf99476a4" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 34699 } health_report { overall_health: UNKNOWN } } }
W20260620 23:05:31.801041 16595 tablet.cc:2369] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:31.837225 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.832121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.12.2.65:38641)
0620 23:05:31.832320 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.832326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.832409 (+ 83us) server_negotiation.cc:407] Connection header received
0620 23:05:31.832622 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.832626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.832703 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.832828 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:31.833194 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.833888 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.835217 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.835604 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.835706 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.836175 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:31.836674 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:31.836801 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.836889 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.836965 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:05:31.844555 16682 raft_consensus.cc:1261] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4 [term 1 FOLLOWER]: Refusing update from remote peer d194bbcb5bf64bfebb75970d19d70c4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:31.844614 16814 raft_consensus.cc:1261] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6 [term 1 FOLLOWER]: Refusing update from remote peer d194bbcb5bf64bfebb75970d19d70c4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:31.845373 16888 consensus_queue.cc:1035] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "21411c22b4744644a18bd93cf99476a4" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 34699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:31.845685 16885 consensus_queue.cc:1035] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "87a9918160cf4cd1bdfb14cad66445a6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:31.855448 16891 mvcc.cc:205] Tried to move back new op lower bound from 7299058613626499072 to 7299058613417246720. Current Snapshot: MvccSnapshot[applied={T|T < 7299058613626499072}]
I20260620 23:05:31.858794 16893 mvcc.cc:205] Tried to move back new op lower bound from 7299058613626499072 to 7299058613417246720. Current Snapshot: MvccSnapshot[applied={T|T < 7299058613626499072}]
I20260620 23:05:31.859053 16892 mvcc.cc:205] Tried to move back new op lower bound from 7299058613626499072 to 7299058613417246720. Current Snapshot: MvccSnapshot[applied={T|T < 7299058613626499072 or (T in {7299058613626499072})}]
I20260620 23:05:31.870579 16405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:31.870838 16405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:31.874224 16405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8149d439bf4c43839551f014e634a59f: Sending DeleteTablet for 3 replicas of tablet 2e901d068ada4ad184326ccfd9dce83d
I20260620 23:05:31.875337 16529 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e901d068ada4ad184326ccfd9dce83d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:31 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:41250
I20260620 23:05:31.875445 16794 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e901d068ada4ad184326ccfd9dce83d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:31 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:42300
I20260620 23:05:31.875967 16662 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e901d068ada4ad184326ccfd9dce83d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:31 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:33618
I20260620 23:05:31.883226 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 16467
I20260620 23:05:31.887689 16901 tablet_replica.cc:330] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: stopping tablet replica
I20260620 23:05:31.888175 16901 raft_consensus.cc:2229] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:31.888731 16901 raft_consensus.cc:2258] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:31.891113 16901 ts_tablet_manager.cc:1905] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:31.895552 16900 tablet_replica.cc:330] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: stopping tablet replica
I20260620 23:05:31.896025 16900 raft_consensus.cc:2229] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:31.896764 16900 raft_consensus.cc:2258] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:31.898869 16900 ts_tablet_manager.cc:1905] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:31.900477 16901 ts_tablet_manager.cc:1918] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:31.900648 16901 log.cc:1196] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/2e901d068ada4ad184326ccfd9dce83d
I20260620 23:05:31.901137 16901 ts_tablet_manager.cc:1939] T 2e901d068ada4ad184326ccfd9dce83d P 87a9918160cf4cd1bdfb14cad66445a6: Deleting consensus metadata
I20260620 23:05:31.903182 16391 catalog_manager.cc:4841] TS 87a9918160cf4cd1bdfb14cad66445a6 (127.12.2.67:32965): tablet 2e901d068ada4ad184326ccfd9dce83d (table test-table [id=718909b6b52c487b898e75e1d712559a]) successfully deleted
I20260620 23:05:31.903658 16902 tablet_replica.cc:330] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: stopping tablet replica
I20260620 23:05:31.904080 16902 raft_consensus.cc:2229] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:31.904585 16902 raft_consensus.cc:2258] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:31.907001 16902 ts_tablet_manager.cc:1905] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:31.907442 16900 ts_tablet_manager.cc:1918] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:31.907672 16900 log.cc:1196] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/2e901d068ada4ad184326ccfd9dce83d
I20260620 23:05:31.908275 16900 ts_tablet_manager.cc:1939] T 2e901d068ada4ad184326ccfd9dce83d P d194bbcb5bf64bfebb75970d19d70c4e: Deleting consensus metadata
I20260620 23:05:31.909983 16392 catalog_manager.cc:4841] TS d194bbcb5bf64bfebb75970d19d70c4e (127.12.2.65:38641): tablet 2e901d068ada4ad184326ccfd9dce83d (table test-table [id=718909b6b52c487b898e75e1d712559a]) successfully deleted
Jun 20 23:05:31 dist-test-slave-7g8g krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:05:31.912856 16902 ts_tablet_manager.cc:1918] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:31.912992 16902 log.cc:1196] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/2e901d068ada4ad184326ccfd9dce83d
I20260620 23:05:31.913491 16902 ts_tablet_manager.cc:1939] T 2e901d068ada4ad184326ccfd9dce83d P 21411c22b4744644a18bd93cf99476a4: Deleting consensus metadata
I20260620 23:05:31.915086 16391 catalog_manager.cc:4841] TS 21411c22b4744644a18bd93cf99476a4 (127.12.2.66:34699): tablet 2e901d068ada4ad184326ccfd9dce83d (table test-table [id=718909b6b52c487b898e75e1d712559a]) successfully deleted
I20260620 23:05:31.919198 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.886413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.12.2.65:38641)
0620 23:05:31.886598 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.886605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.901495 (+ 14890us) server_negotiation.cc:407] Connection header received
0620 23:05:31.902025 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.902031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.902110 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.902179 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:31.904513 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.905733 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.907444 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.908089 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.908196 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.913252 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:31.913265 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:31.913287 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:31.913331 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:31.916499 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:31.917268 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:31.917275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:31.917277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:31.917414 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:31.917789 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:31.917792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:31.917794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:31.918090 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:31.918314 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.918595 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.918751 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:05:31.919884 16589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:32.027235 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 16467
I20260620 23:05:32.041245 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 16600
Jun 20 23:05:32 dist-test-slave-7g8g krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:05:32.055090 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:32.041996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.12.2.66:34699)
0620 23:05:32.042167 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:32.042172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:32.042192 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:05:32.042357 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:32.042360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:32.042438 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:32.042531 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:32.043778 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:32.044573 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:32.045756 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:32.046126 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:32.046231 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:32.049977 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:32.049990 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:32.050006 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:32.050041 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:32.052658 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:32.053252 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:32.053259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:32.053262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:32.053377 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:32.053717 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:32.053720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:32.053722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:32.054042 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:32.054242 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:32.054480 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 23:05:32.054694 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:05:32.055750 16722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:32.141870 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 16600
I20260620 23:05:32.154417 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 16732
Jun 20 23:05:32 dist-test-slave-7g8g krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:05:32.168303 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:32.155159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.12.2.67:32965)
0620 23:05:32.155329 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:32.155334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:32.155352 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:05:32.155453 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:32.155458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:32.155519 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:32.155596 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:32.156737 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:32.157514 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:32.158921 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:32.159300 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:32.159390 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:32.163228 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:32.163243 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:32.163259 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:32.163298 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:32.165785 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:32.166418 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:32.166435 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:32.166437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:32.166580 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:32.166873 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:32.166876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:32.166878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:32.167336 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:32.167547 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:32.167809 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 23:05:32.167963 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:05:32.168874 16854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:32.254424 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 16732
I20260620 23:05:32.266628 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 16375
I20260620 23:05:32.267735 16436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:32.372292 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 16375
2026-06-20T23:05:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3937 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T23:05:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:32Z Disabled control of system clock
I20260620 23:05:32.430420 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:42581
--webserver_interface=127.12.2.126
--webserver_port=0
--builtin_ntp_servers=127.12.2.84:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:42581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:05:32.731108 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:32.731369 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:32.731454 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:32.738377 16911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:32.738476 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:32.738510 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:32.738569 16911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:32.738610 16911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:32.747390 16911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:42581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:42581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:32.748597 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:32.750202 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:32.760888 16917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:32.760955 16916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:32.760928 16919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:32.761878 16911 server_base.cc:1034] running on GCE node
I20260620 23:05:32.762574 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:32.763892 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:32.765074 16911 hybrid_clock.cc:648] HybridClock initialized: now 1781996732765035 us; error 35 us; skew 500 ppm
I20260620 23:05:32.765377 16911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:05:32.767414 16911 webserver.cc:460] Webserver started at http://127.12.2.126:42877/ using document root <none> and password file <none>
I20260620 23:05:32.768038 16911 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:32.768132 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:32.768435 16911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:32.771189 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "24ea30c1f60d4fa6acc48eebd941f8af"
format_stamp: "Formatted at 2026-06-20 23:05:32 on dist-test-slave-7g8g"
server_key: "76a77b56e1001489f3c138d30376bf94"
server_key_iv: "85407f9c42d7181fa9468335603e969a"
server_key_version: "encryptionkey@0"
I20260620 23:05:32.771935 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "24ea30c1f60d4fa6acc48eebd941f8af"
format_stamp: "Formatted at 2026-06-20 23:05:32 on dist-test-slave-7g8g"
server_key: "76a77b56e1001489f3c138d30376bf94"
server_key_iv: "85407f9c42d7181fa9468335603e969a"
server_key_version: "encryptionkey@0"
I20260620 23:05:32.776777 16911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:05:32.779929 16925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:32.781133 16911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:05:32.781286 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "24ea30c1f60d4fa6acc48eebd941f8af"
format_stamp: "Formatted at 2026-06-20 23:05:32 on dist-test-slave-7g8g"
server_key: "76a77b56e1001489f3c138d30376bf94"
server_key_iv: "85407f9c42d7181fa9468335603e969a"
server_key_version: "encryptionkey@0"
I20260620 23:05:32.781468 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:32.787101 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:32.787883 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:32.788178 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:32.805027 16911 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:42581
I20260620 23:05:32.805038 16976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:42581 every 8 connection(s)
I20260620 23:05:32.806253 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:32.809829 16977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:32.815954 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 16911
I20260620 23:05:32.816143 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:32.816927 12297 external_mini_cluster.cc:1428] Setting key 5c8d517ccb2a3ea3d9eb12f9295c95be
I20260620 23:05:32.820335 16977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af: Bootstrap starting.
I20260620 23:05:32.823544 16977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:32.824630 16977 log.cc:824] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:32.824898 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:32.818960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.12.2.126:42581)
0620 23:05:32.819846 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:32.819856 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:32.819896 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:05:32.821498 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:32.821520 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:32.821946 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:32.822267 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:32.823333 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:32.823339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:32.823351 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:32.823365 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:32.823421 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:32.823440 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:32.823570 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:32.823751 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 23:05:32.824010 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":288,"threads_started":1}
I20260620 23:05:32.827427 16977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af: No bootstrap required, opened a new log
I20260620 23:05:32.834065 16977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ea30c1f60d4fa6acc48eebd941f8af" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42581 } }
I20260620 23:05:32.834390 16977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:32.834473 16977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ea30c1f60d4fa6acc48eebd941f8af, State: Initialized, Role: FOLLOWER
I20260620 23:05:32.835002 16977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ea30c1f60d4fa6acc48eebd941f8af" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42581 } }
I20260620 23:05:32.835211 16977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:32.835322 16977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:32.835465 16977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:32.837888 16977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ea30c1f60d4fa6acc48eebd941f8af" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42581 } }
I20260620 23:05:32.838338 16977 leader_election.cc:302] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24ea30c1f60d4fa6acc48eebd941f8af; no voters:
I20260620 23:05:32.838852 16977 leader_election.cc:288] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:32.838992 16982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:32.840384 16982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [term 1 LEADER]: Becoming Leader. State: Replica: 24ea30c1f60d4fa6acc48eebd941f8af, State: Running, Role: LEADER
I20260620 23:05:32.840718 16977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:32.840891 16982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ea30c1f60d4fa6acc48eebd941f8af" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42581 } }
I20260620 23:05:32.843381 16984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ea30c1f60d4fa6acc48eebd941f8af. Latest consensus state: current_term: 1 leader_uuid: "24ea30c1f60d4fa6acc48eebd941f8af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ea30c1f60d4fa6acc48eebd941f8af" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42581 } } }
I20260620 23:05:32.843740 16984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:32.844331 16983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24ea30c1f60d4fa6acc48eebd941f8af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ea30c1f60d4fa6acc48eebd941f8af" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42581 } } }
I20260620 23:05:32.844636 16983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:32.845994 16993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:32.850539 16993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:32.858114 16993 catalog_manager.cc:1347] Generated new cluster ID: d91eeeb08ace4c93a7fdad3bfe6c99cb
I20260620 23:05:32.858224 16993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:32.879528 16993 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:32.879915 16993 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:32.885820 16993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af: Generated new TSK 0
I20260620 23:05:32.886610 16993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:32.898564 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42581
--builtin_ntp_servers=127.12.2.84:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:05:33.210582 17001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:33.210855 17001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:33.210973 17001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:33.218196 17001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:33.218309 17001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:33.218422 17001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:33.227636 17001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:33.228857 17001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:33.230494 17001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:33.243283 17007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.243314 17006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.243381 17009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:33.244064 17001 server_base.cc:1034] running on GCE node
I20260620 23:05:33.244673 17001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:33.245592 17001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:33.246827 17001 hybrid_clock.cc:648] HybridClock initialized: now 1781996733246762 us; error 51 us; skew 500 ppm
I20260620 23:05:33.247166 17001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:05:33.249318 17001 webserver.cc:460] Webserver started at http://127.12.2.65:37627/ using document root <none> and password file <none>
I20260620 23:05:33.249996 17001 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:33.250101 17001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:33.250406 17001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:33.253244 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "c03df510262b40a28696fc7260ef660e"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-7g8g"
server_key: "a2ed5e345e034a73551e1908b2e61e5b"
server_key_iv: "0b02b77cd55526e979c1e9858d96768a"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.253970 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "c03df510262b40a28696fc7260ef660e"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-7g8g"
server_key: "a2ed5e345e034a73551e1908b2e61e5b"
server_key_iv: "0b02b77cd55526e979c1e9858d96768a"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.259027 17001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:05:33.262270 17015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.263541 17001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:33.263703 17001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "c03df510262b40a28696fc7260ef660e"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-7g8g"
server_key: "a2ed5e345e034a73551e1908b2e61e5b"
server_key_iv: "0b02b77cd55526e979c1e9858d96768a"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.263887 17001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:33.269217 17001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:33.269930 17001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:33.270159 17001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:33.270979 17001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:33.272223 17001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:33.272320 17001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.272418 17001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:33.272485 17001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.302219 17001 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:45647
I20260620 23:05:33.302273 17127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:45647 every 8 connection(s)
I20260620 23:05:33.303926 17001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:33.306875 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 17001
I20260620 23:05:33.307076 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:33.307706 12297 external_mini_cluster.cc:1428] Setting key 88c7741e742960597f34332298cc3471
I20260620 23:05:33.310211 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42581
--builtin_ntp_servers=127.12.2.84:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 23:05:33.312371 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.307022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:58235 (local address 127.12.2.126:42581)
0620 23:05:33.307211 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.307218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.308721 (+ 1503us) server_negotiation.cc:407] Connection header received
0620 23:05:33.310354 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.310360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.310420 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.310513 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:33.311766 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:33.311771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:33.311776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:33.311796 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:33.311831 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:33.311844 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:33.311954 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.312073 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.312125 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:05:33.313226 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.307541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:42581 (local address 127.12.2.65:58235)
0620 23:05:33.308569 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:33.308612 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:33.310092 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:33.310713 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:33.310720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:33.310817 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:33.311449 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:33.311451 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:33.311489 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:33.311604 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:33.311922 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:33.311927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:33.311932 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:05:33.312121 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 23:05:33.312312 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":303,"threads_started":1}
I20260620 23:05:33.314604 17128 heartbeater.cc:347] Connected to a master server at 127.12.2.126:42581
I20260620 23:05:33.314870 17128 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:33.315567 17128 heartbeater.cc:511] Master 127.12.2.126:42581 requested a full tablet report, sending...
I20260620 23:05:33.317726 16942 ts_manager.cc:194] Registered new tserver with Master: c03df510262b40a28696fc7260ef660e (127.12.2.65:45647)
W20260620 23:05:33.613814 17131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:33.614073 17131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:33.614164 17131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:33.621120 17131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:33.621207 17131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:33.621312 17131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:33.629925 17131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:33.631093 17131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:33.632732 17131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:33.644829 17139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.644927 17137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.644852 17136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:33.645537 17131 server_base.cc:1034] running on GCE node
I20260620 23:05:33.646071 17131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:33.646965 17131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:33.648175 17131 hybrid_clock.cc:648] HybridClock initialized: now 1781996733648117 us; error 35 us; skew 500 ppm
I20260620 23:05:33.648468 17131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:05:33.650437 17131 webserver.cc:460] Webserver started at http://127.12.2.66:33111/ using document root <none> and password file <none>
I20260620 23:05:33.651101 17131 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:33.651204 17131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:33.651497 17131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:33.654072 17131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "7def6d99a9654b8aa05c4db421204e7a"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-7g8g"
server_key: "5f8e90dcc3961aea257c9cb43d130e0d"
server_key_iv: "1fe257bd7836c3f0c91d6028b60ffe1e"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.654824 17131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "7def6d99a9654b8aa05c4db421204e7a"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-7g8g"
server_key: "5f8e90dcc3961aea257c9cb43d130e0d"
server_key_iv: "1fe257bd7836c3f0c91d6028b60ffe1e"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.659529 17131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:05:33.662643 17145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.663856 17131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:05:33.663995 17131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "7def6d99a9654b8aa05c4db421204e7a"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-7g8g"
server_key: "5f8e90dcc3961aea257c9cb43d130e0d"
server_key_iv: "1fe257bd7836c3f0c91d6028b60ffe1e"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.664186 17131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:33.669289 17131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:33.669976 17131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:33.670204 17131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:33.671077 17131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:33.672349 17131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:33.672432 17131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.672535 17131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:33.672600 17131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.700412 17131 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:44185
I20260620 23:05:33.700457 17257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:44185 every 8 connection(s)
I20260620 23:05:33.701646 17131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:33.708169 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 17131
I20260620 23:05:33.708355 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:33.708895 12297 external_mini_cluster.cc:1428] Setting key 75a4baf6e9bc30c00f56b69e17392427
I20260620 23:05:33.709307 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.704043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:43193 (local address 127.12.2.126:42581)
0620 23:05:33.704201 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.704205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.705608 (+ 1403us) server_negotiation.cc:407] Connection header received
0620 23:05:33.707240 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.707246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.707310 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.707393 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:33.708658 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:33.708664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:33.708668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:33.708687 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:33.708718 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:33.708730 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:33.708835 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.709038 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.709091 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:05:33.710230 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.704582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:42581 (local address 127.12.2.66:43193)
0620 23:05:33.705460 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:33.705499 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:33.706986 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:33.707621 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:33.707629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:33.707720 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:33.708339 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:33.708343 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:33.708391 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:33.708495 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:33.708823 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:33.708832 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:33.708838 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:05:33.709094 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 23:05:33.709346 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":259,"threads_started":1}
I20260620 23:05:33.711404 17258 heartbeater.cc:347] Connected to a master server at 127.12.2.126:42581
I20260620 23:05:33.711621 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42581
--builtin_ntp_servers=127.12.2.84:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 23:05:33.711695 17258 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:33.712278 17258 heartbeater.cc:511] Master 127.12.2.126:42581 requested a full tablet report, sending...
I20260620 23:05:33.713624 16942 ts_manager.cc:194] Registered new tserver with Master: 7def6d99a9654b8aa05c4db421204e7a (127.12.2.66:44185)
W20260620 23:05:34.014284 17261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:34.014611 17261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:34.014755 17261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:34.022264 17261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:34.022398 17261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:34.022567 17261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:34.031917 17261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:34.033228 17261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:34.035159 17261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:34.048501 17266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:34.048547 17269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:34.048601 17267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:34.051595 17261 server_base.cc:1034] running on GCE node
I20260620 23:05:34.052174 17261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:34.053066 17261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:34.054267 17261 hybrid_clock.cc:648] HybridClock initialized: now 1781996734054222 us; error 35 us; skew 500 ppm
I20260620 23:05:34.054554 17261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:05:34.056734 17261 webserver.cc:460] Webserver started at http://127.12.2.67:46027/ using document root <none> and password file <none>
I20260620 23:05:34.057404 17261 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:34.057487 17261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:34.057761 17261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:34.060596 17261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "2329842e7d63412abf855f12b79d8969"
format_stamp: "Formatted at 2026-06-20 23:05:34 on dist-test-slave-7g8g"
server_key: "76c8ad2319a1f70bf40869944120892a"
server_key_iv: "2ce5d0c9d61f2acd4769b7afeb0e68ed"
server_key_version: "encryptionkey@0"
I20260620 23:05:34.061328 17261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "2329842e7d63412abf855f12b79d8969"
format_stamp: "Formatted at 2026-06-20 23:05:34 on dist-test-slave-7g8g"
server_key: "76c8ad2319a1f70bf40869944120892a"
server_key_iv: "2ce5d0c9d61f2acd4769b7afeb0e68ed"
server_key_version: "encryptionkey@0"
I20260620 23:05:34.066313 17261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:05:34.069775 17275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:34.071134 17261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:34.071266 17261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "2329842e7d63412abf855f12b79d8969"
format_stamp: "Formatted at 2026-06-20 23:05:34 on dist-test-slave-7g8g"
server_key: "76c8ad2319a1f70bf40869944120892a"
server_key_iv: "2ce5d0c9d61f2acd4769b7afeb0e68ed"
server_key_version: "encryptionkey@0"
I20260620 23:05:34.071460 17261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:34.077064 17261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:34.077829 17261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:34.078073 17261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:34.079016 17261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:34.080441 17261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:34.080523 17261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:34.080605 17261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:34.080677 17261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:34.109956 17261 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:36431
I20260620 23:05:34.110004 17387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:36431 every 8 connection(s)
I20260620 23:05:34.111248 17261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:34.118654 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.113686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:55467 (local address 127.12.2.126:42581)
0620 23:05:34.113858 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.113862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.115197 (+ 1335us) server_negotiation.cc:407] Connection header received
0620 23:05:34.116783 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.116787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.116858 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.116944 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.118137 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.118140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.118143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.118155 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.118178 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.118188 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.118274 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.118400 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.118444 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:05:34.119167 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 17261
I20260620 23:05:34.119289 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:34.119654 12297 external_mini_cluster.cc:1428] Setting key 5ce28709338bdd21de2243be6b0aa300
I20260620 23:05:34.120898 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.114164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:42581 (local address 127.12.2.67:55467)
0620 23:05:34.115057 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.115096 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.116508 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.117137 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.117145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.117233 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:34.117885 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:34.117888 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:34.117923 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:34.118007 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:34.118258 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:34.118264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:34.118268 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.118456 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.118644 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":247,"threads_started":1}
I20260620 23:05:34.122229 17388 heartbeater.cc:347] Connected to a master server at 127.12.2.126:42581
I20260620 23:05:34.122512 17388 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:34.123062 17388 heartbeater.cc:511] Master 127.12.2.126:42581 requested a full tablet report, sending...
I20260620 23:05:34.124344 16942 ts_manager.cc:194] Registered new tserver with Master: 2329842e7d63412abf855f12b79d8969 (127.12.2.67:36431)
I20260620 23:05:34.134686 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:05:34.141114 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.138747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.12.2.126:42581)
0620 23:05:34.138910 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.138916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.139338 (+ 422us) server_negotiation.cc:407] Connection header received
0620 23:05:34.139491 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.139495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.139542 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.139627 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.140454 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.140459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.140462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.140475 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.140505 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.140516 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.140614 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.140745 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.140802 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:05:34.144999 16942 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:34.154886 16942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:34.171989 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.166816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.12.2.65:45647)
0620 23:05:34.167374 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.167382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.168403 (+ 1021us) server_negotiation.cc:407] Connection header received
0620 23:05:34.168483 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.168492 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.168822 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.168954 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.170986 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.170996 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.171009 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.171029 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.171096 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.171117 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.171216 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.171536 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.171642 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":242,"threads_started":1}
I20260620 23:05:34.173043 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.166601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:45647 (local address 127.0.0.1:57178)
0620 23:05:34.168019 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.168058 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.168227 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.169811 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.169819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.169909 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:34.170649 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:34.170652 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:34.170687 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:34.170783 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:34.171323 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:34.171332 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:34.171339 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.172520 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.172609 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"thread_start_us":2649,"threads_started":1}
I20260620 23:05:34.173653 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.167237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:36431 (local address 127.0.0.1:45100)
0620 23:05:34.169501 (+ 2264us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.169518 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.169617 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.171151 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.171156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.171180 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:34.172721 (+ 1541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:34.172723 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:34.172736 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:34.172775 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:34.173112 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:34.173116 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:34.173121 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.173234 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.173280 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2132,"thread_start_us":363,"threads_started":1}
I20260620 23:05:34.173682 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.167654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.12.2.67:36431)
0620 23:05:34.170634 (+ 2980us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.170643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.170677 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:05:34.170742 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.170750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.170990 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.171127 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.172923 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.172930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.172941 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.172955 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.172998 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.173012 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.173173 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.173235 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.173326 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2556,"thread_start_us":2400,"threads_started":1}
I20260620 23:05:34.175603 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.169850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.12.2.66:44185)
0620 23:05:34.172699 (+ 2849us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.172707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.172734 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:05:34.172798 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.172805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.173002 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.173150 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.174859 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.174867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.174876 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.174890 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.174967 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.174993 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.175145 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.175204 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.175308 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2696,"thread_start_us":2583,"threads_started":1}
I20260620 23:05:34.175810 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.170214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:44185 (local address 127.0.0.1:36128)
0620 23:05:34.170717 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.170733 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.170828 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.174018 (+ 3190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.174023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.174046 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:34.174647 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:34.174650 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:34.174667 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:34.174706 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:34.175075 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:34.175080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:34.175084 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.175195 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.175243 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":390,"threads_started":1}
I20260620 23:05:34.177237 17063 tablet_service.cc:1467] Processing CreateTablet for tablet ddf59c0e82ae449c96f15fb8c22dcd7f (DEFAULT_TABLE table=test-table [id=558bfc8334af438eaa3f703da9b924b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:34.177235 17323 tablet_service.cc:1467] Processing CreateTablet for tablet ddf59c0e82ae449c96f15fb8c22dcd7f (DEFAULT_TABLE table=test-table [id=558bfc8334af438eaa3f703da9b924b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:34.178849 17323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf59c0e82ae449c96f15fb8c22dcd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:34.178849 17063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf59c0e82ae449c96f15fb8c22dcd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:34.179621 17193 tablet_service.cc:1467] Processing CreateTablet for tablet ddf59c0e82ae449c96f15fb8c22dcd7f (DEFAULT_TABLE table=test-table [id=558bfc8334af438eaa3f703da9b924b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:34.180749 17193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf59c0e82ae449c96f15fb8c22dcd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:34.192198 17406 tablet_bootstrap.cc:492] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a: Bootstrap starting.
I20260620 23:05:34.194322 17407 tablet_bootstrap.cc:492] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: Bootstrap starting.
I20260620 23:05:34.197327 17406 tablet_bootstrap.cc:654] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:34.197628 17407 tablet_bootstrap.cc:654] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:34.198228 17408 tablet_bootstrap.cc:492] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: Bootstrap starting.
I20260620 23:05:34.198573 17406 log.cc:824] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:34.198659 17407 log.cc:824] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:34.201370 17407 tablet_bootstrap.cc:492] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: No bootstrap required, opened a new log
I20260620 23:05:34.201642 17407 ts_tablet_manager.cc:1397] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 23:05:34.201680 17406 tablet_bootstrap.cc:492] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a: No bootstrap required, opened a new log
I20260620 23:05:34.201927 17406 ts_tablet_manager.cc:1397] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 23:05:34.202071 17408 tablet_bootstrap.cc:654] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:34.203325 17408 log.cc:824] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:34.206295 17408 tablet_bootstrap.cc:492] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: No bootstrap required, opened a new log
I20260620 23:05:34.206542 17408 ts_tablet_manager.cc:1397] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:05:34.209041 17407 raft_consensus.cc:348] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03df510262b40a28696fc7260ef660e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 45647 } } peers { permanent_uuid: "7def6d99a9654b8aa05c4db421204e7a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44185 } } peers { permanent_uuid: "2329842e7d63412abf855f12b79d8969" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36431 } }
I20260620 23:05:34.209388 17407 raft_consensus.cc:374] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:34.209471 17407 raft_consensus.cc:729] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03df510262b40a28696fc7260ef660e, State: Initialized, Role: FOLLOWER
I20260620 23:05:34.210044 17407 consensus_queue.cc:260] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03df510262b40a28696fc7260ef660e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 45647 } } peers { permanent_uuid: "7def6d99a9654b8aa05c4db421204e7a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44185 } } peers { permanent_uuid: "2329842e7d63412abf855f12b79d8969" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36431 } }
I20260620 23:05:34.211449 17406 raft_consensus.cc:348] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03df510262b40a28696fc7260ef660e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 45647 } } peers { permanent_uuid: "7def6d99a9654b8aa05c4db421204e7a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44185 } } peers { permanent_uuid: "2329842e7d63412abf855f12b79d8969" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36431 } }
I20260620 23:05:34.211867 17406 raft_consensus.cc:374] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:34.211958 17406 raft_consensus.cc:729] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7def6d99a9654b8aa05c4db421204e7a, State: Initialized, Role: FOLLOWER
I20260620 23:05:34.212528 17406 consensus_queue.cc:260] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03df510262b40a28696fc7260ef660e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 45647 } } peers { permanent_uuid: "7def6d99a9654b8aa05c4db421204e7a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44185 } } peers { permanent_uuid: "2329842e7d63412abf855f12b79d8969" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36431 } }
I20260620 23:05:34.213587 17258 heartbeater.cc:503] Master 127.12.2.126:42581 was elected leader, sending a full tablet report...
I20260620 23:05:34.214236 17406 ts_tablet_manager.cc:1428] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260620 23:05:34.216807 17408 raft_consensus.cc:348] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03df510262b40a28696fc7260ef660e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 45647 } } peers { permanent_uuid: "7def6d99a9654b8aa05c4db421204e7a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44185 } } peers { permanent_uuid: "2329842e7d63412abf855f12b79d8969" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36431 } }
I20260620 23:05:34.217151 17408 raft_consensus.cc:374] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:34.217264 17408 raft_consensus.cc:729] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2329842e7d63412abf855f12b79d8969, State: Initialized, Role: FOLLOWER
I20260620 23:05:34.217944 17128 heartbeater.cc:503] Master 127.12.2.126:42581 was elected leader, sending a full tablet report...
I20260620 23:05:34.218590 17407 ts_tablet_manager.cc:1428] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260620 23:05:34.218997 17408 consensus_queue.cc:260] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03df510262b40a28696fc7260ef660e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 45647 } } peers { permanent_uuid: "7def6d99a9654b8aa05c4db421204e7a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44185 } } peers { permanent_uuid: "2329842e7d63412abf855f12b79d8969" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36431 } }
I20260620 23:05:34.220106 17388 heartbeater.cc:503] Master 127.12.2.126:42581 was elected leader, sending a full tablet report...
I20260620 23:05:34.220781 17408 ts_tablet_manager.cc:1428] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
W20260620 23:05:34.306524 17129 tablet.cc:2369] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:34.350387 17414 raft_consensus.cc:482] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:34.350668 17414 raft_consensus.cc:504] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03df510262b40a28696fc7260ef660e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 45647 } } peers { permanent_uuid: "7def6d99a9654b8aa05c4db421204e7a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44185 } } peers { permanent_uuid: "2329842e7d63412abf855f12b79d8969" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36431 } }
I20260620 23:05:34.352293 17414 leader_election.cc:288] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c03df510262b40a28696fc7260ef660e (127.12.2.65:45647), 7def6d99a9654b8aa05c4db421204e7a (127.12.2.66:44185)
I20260620 23:05:34.354624 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.352725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:51271 (local address 127.12.2.65:45647)
0620 23:05:34.352911 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.352918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.352940 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:05:34.353047 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.353052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.353112 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.353218 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.354075 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.354080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.354084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.354103 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.354140 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.354155 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.354265 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.354349 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.354406 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:05:34.354630 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.352625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:45647 (local address 127.12.2.67:51271)
0620 23:05:34.352815 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.352834 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.352923 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.353266 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.353270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.353296 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:34.353865 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:34.353868 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:34.353886 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:34.353933 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:34.354251 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:34.354257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:34.354262 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.354346 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.354393 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:05:34.355590 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.353154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:49031 (local address 127.12.2.66:44185)
0620 23:05:34.353329 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.353336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.353858 (+ 522us) server_negotiation.cc:407] Connection header received
0620 23:05:34.353934 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.353939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.353998 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.354086 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.355094 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.355100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.355105 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.355124 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.355156 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.355167 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.355283 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.355373 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.355415 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:05:34.355547 17083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf59c0e82ae449c96f15fb8c22dcd7f" candidate_uuid: "2329842e7d63412abf855f12b79d8969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03df510262b40a28696fc7260ef660e" is_pre_election: true
I20260620 23:05:34.355841 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.353012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:44185 (local address 127.12.2.67:49031)
0620 23:05:34.353761 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.353775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.353860 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.354124 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.354129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.354150 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:34.354880 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:34.354883 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:34.354898 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:34.354968 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:34.355271 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:34.355275 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:34.355280 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.355393 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.355440 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":464,"threads_started":1}
I20260620 23:05:34.356117 17083 raft_consensus.cc:2454] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329842e7d63412abf855f12b79d8969 in term 0.
I20260620 23:05:34.356694 17213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf59c0e82ae449c96f15fb8c22dcd7f" candidate_uuid: "2329842e7d63412abf855f12b79d8969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7def6d99a9654b8aa05c4db421204e7a" is_pre_election: true
I20260620 23:05:34.356817 17278 leader_election.cc:302] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329842e7d63412abf855f12b79d8969, c03df510262b40a28696fc7260ef660e; no voters:
I20260620 23:05:34.357165 17414 raft_consensus.cc:2781] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:34.357193 17213 raft_consensus.cc:2454] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2329842e7d63412abf855f12b79d8969 in term 0.
I20260620 23:05:34.357306 17414 raft_consensus.cc:482] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:34.357401 17414 raft_consensus.cc:3037] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:34.359555 17414 raft_consensus.cc:504] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03df510262b40a28696fc7260ef660e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 45647 } } peers { permanent_uuid: "7def6d99a9654b8aa05c4db421204e7a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44185 } } peers { permanent_uuid: "2329842e7d63412abf855f12b79d8969" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36431 } }
I20260620 23:05:34.360054 17414 leader_election.cc:288] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [CANDIDATE]: Term 1 election: Requested vote from peers c03df510262b40a28696fc7260ef660e (127.12.2.65:45647), 7def6d99a9654b8aa05c4db421204e7a (127.12.2.66:44185)
I20260620 23:05:34.360586 17213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf59c0e82ae449c96f15fb8c22dcd7f" candidate_uuid: "2329842e7d63412abf855f12b79d8969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7def6d99a9654b8aa05c4db421204e7a"
I20260620 23:05:34.360612 17083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf59c0e82ae449c96f15fb8c22dcd7f" candidate_uuid: "2329842e7d63412abf855f12b79d8969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03df510262b40a28696fc7260ef660e"
I20260620 23:05:34.360790 17213 raft_consensus.cc:3037] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:34.360790 17083 raft_consensus.cc:3037] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:05:34.362665 17389 tablet.cc:2369] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:34.363854 17083 raft_consensus.cc:2454] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329842e7d63412abf855f12b79d8969 in term 1.
I20260620 23:05:34.363854 17213 raft_consensus.cc:2454] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2329842e7d63412abf855f12b79d8969 in term 1.
I20260620 23:05:34.364213 17278 leader_election.cc:302] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2329842e7d63412abf855f12b79d8969, 7def6d99a9654b8aa05c4db421204e7a; no voters:
I20260620 23:05:34.364446 17414 raft_consensus.cc:2781] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:34.364892 17414 raft_consensus.cc:686] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 1 LEADER]: Becoming Leader. State: Replica: 2329842e7d63412abf855f12b79d8969, State: Running, Role: LEADER
I20260620 23:05:34.365391 17414 consensus_queue.cc:237] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03df510262b40a28696fc7260ef660e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 45647 } } peers { permanent_uuid: "7def6d99a9654b8aa05c4db421204e7a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44185 } } peers { permanent_uuid: "2329842e7d63412abf855f12b79d8969" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36431 } }
I20260620 23:05:34.369920 16940 catalog_manager.cc:5495] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2329842e7d63412abf855f12b79d8969 (127.12.2.67). New cstate: current_term: 1 leader_uuid: "2329842e7d63412abf855f12b79d8969" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03df510262b40a28696fc7260ef660e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 45647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7def6d99a9654b8aa05c4db421204e7a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2329842e7d63412abf855f12b79d8969" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 36431 } health_report { overall_health: HEALTHY } } }
I20260620 23:05:34.415650 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.413643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.12.2.67:36431)
0620 23:05:34.413847 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.413852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.413873 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:05:34.414074 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.414078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.414151 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.414229 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.415160 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.415164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.415167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.415180 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.415209 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.415220 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.415311 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.415364 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.415408 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:05:34.424165 17083 raft_consensus.cc:1261] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e [term 1 FOLLOWER]: Refusing update from remote peer 2329842e7d63412abf855f12b79d8969: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:34.424288 17213 raft_consensus.cc:1261] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a [term 1 FOLLOWER]: Refusing update from remote peer 2329842e7d63412abf855f12b79d8969: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:34.425110 17414 consensus_queue.cc:1035] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03df510262b40a28696fc7260ef660e" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 45647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:34.425403 17416 consensus_queue.cc:1035] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7def6d99a9654b8aa05c4db421204e7a" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:34.444342 17419 mvcc.cc:205] Tried to move back new op lower bound from 7299058624190406656 to 7299058623964295168. Current Snapshot: MvccSnapshot[applied={T|T < 7299058624190406656 or (T in {7299058624190406656})}]
I20260620 23:05:34.453709 16940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:34.453982 16940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:34.460692 17063 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf59c0e82ae449c96f15fb8c22dcd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:34 UTC) from {username='slave'} at 127.0.0.1:57178
I20260620 23:05:34.462023 17323 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf59c0e82ae449c96f15fb8c22dcd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:34 UTC) from {username='slave'} at 127.0.0.1:45100
I20260620 23:05:34.462764 17430 tablet_replica.cc:330] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: stopping tablet replica
I20260620 23:05:34.463172 17430 raft_consensus.cc:2229] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:34.463896 17430 raft_consensus.cc:2258] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:34.463944 17193 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf59c0e82ae449c96f15fb8c22dcd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:34 UTC) from {username='slave'} at 127.0.0.1:36128
I20260620 23:05:34.464977 16940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24ea30c1f60d4fa6acc48eebd941f8af: Sending DeleteTablet for 3 replicas of tablet ddf59c0e82ae449c96f15fb8c22dcd7f
I20260620 23:05:34.465708 17192 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf59c0e82ae449c96f15fb8c22dcd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:34 UTC) from {username='slave'} at 127.0.0.1:36128
I20260620 23:05:34.465896 17062 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf59c0e82ae449c96f15fb8c22dcd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:34 UTC) from {username='slave'} at 127.0.0.1:57178
I20260620 23:05:34.466493 17323 tablet_service.cc:1514] Processing DeleteTablet for tablet ddf59c0e82ae449c96f15fb8c22dcd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:34 UTC) from {username='slave'} at 127.0.0.1:45100
I20260620 23:05:34.467370 17429 tablet_replica.cc:330] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: stopping tablet replica
I20260620 23:05:34.467717 17429 raft_consensus.cc:2229] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:34.468168 17429 raft_consensus.cc:2258] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:34.469621 17429 ts_tablet_manager.cc:1905] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:34.471870 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 17001
I20260620 23:05:34.472465 17430 ts_tablet_manager.cc:1905] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:34.475680 17429 ts_tablet_manager.cc:1918] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:34.475823 17429 log.cc:1196] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/ddf59c0e82ae449c96f15fb8c22dcd7f
I20260620 23:05:34.476313 17429 ts_tablet_manager.cc:1939] T ddf59c0e82ae449c96f15fb8c22dcd7f P c03df510262b40a28696fc7260ef660e: Deleting consensus metadata
W20260620 23:05:34.478483 16928 catalog_manager.cc:4808] TS c03df510262b40a28696fc7260ef660e (127.12.2.65:45647): delete failed for tablet ddf59c0e82ae449c96f15fb8c22dcd7f because the tablet was not found. No further retry: Not found: Tablet not found: ddf59c0e82ae449c96f15fb8c22dcd7f
I20260620 23:05:34.479218 17430 ts_tablet_manager.cc:1918] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:34.479367 17430 log.cc:1196] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/ddf59c0e82ae449c96f15fb8c22dcd7f
I20260620 23:05:34.479776 17430 ts_tablet_manager.cc:1939] T ddf59c0e82ae449c96f15fb8c22dcd7f P 2329842e7d63412abf855f12b79d8969: Deleting consensus metadata
W20260620 23:05:34.481426 16929 catalog_manager.cc:4808] TS 2329842e7d63412abf855f12b79d8969 (127.12.2.67:36431): delete failed for tablet ddf59c0e82ae449c96f15fb8c22dcd7f because the tablet was not found. No further retry: Not found: Tablet not found: ddf59c0e82ae449c96f15fb8c22dcd7f
I20260620 23:05:34.481644 17431 tablet_replica.cc:330] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a: stopping tablet replica
I20260620 23:05:34.481967 17431 raft_consensus.cc:2229] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:34.482347 17431 raft_consensus.cc:2258] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:34.485103 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.473251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.12.2.65:45647)
0620 23:05:34.473417 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.473423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.483122 (+ 9699us) server_negotiation.cc:407] Connection header received
0620 23:05:34.483310 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.483316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.483405 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.483502 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.484410 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.484416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.484421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.484440 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.484495 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.484512 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.484632 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.484776 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.484834 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:05:34.485962 17123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:34.486538 17431 ts_tablet_manager.cc:1905] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:34.491724 17431 ts_tablet_manager.cc:1918] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:34.491827 17431 log.cc:1196] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/ddf59c0e82ae449c96f15fb8c22dcd7f
I20260620 23:05:34.492172 17431 ts_tablet_manager.cc:1939] T ddf59c0e82ae449c96f15fb8c22dcd7f P 7def6d99a9654b8aa05c4db421204e7a: Deleting consensus metadata
W20260620 23:05:34.493764 16928 catalog_manager.cc:4808] TS 7def6d99a9654b8aa05c4db421204e7a (127.12.2.66:44185): delete failed for tablet ddf59c0e82ae449c96f15fb8c22dcd7f because the tablet was not found. No further retry: Not found: Tablet not found: ddf59c0e82ae449c96f15fb8c22dcd7f
I20260620 23:05:34.591966 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 17001
I20260620 23:05:34.611657 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 17131
I20260620 23:05:34.614399 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.612371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36136 (local address 127.12.2.66:44185)
0620 23:05:34.612533 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.612539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.612560 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:05:34.612744 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.612748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.612831 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.612910 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.613815 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.613820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.613823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.613838 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.613867 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.613879 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.613989 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.614095 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.614153 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:05:34.615015 17253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:34.698292 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 17131
I20260620 23:05:34.710899 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 17261
I20260620 23:05:34.713763 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.711902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.12.2.67:36431)
0620 23:05:34.712082 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.712087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.712101 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:05:34.712173 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.712177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.712224 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.712312 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.713134 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.713140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.713144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.713159 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.713188 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.713199 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.713316 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.713406 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.713461 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:05:34.714453 17383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:34.835031 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 17261
I20260620 23:05:34.850114 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 16911
I20260620 23:05:34.851181 16972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:34.953086 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 16911
2026-06-20T23:05:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2579 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 23:05:34.988466 12297 mini_oidc.cc:152] Starting JWKS server
I20260620 23:05:34.990902 12297 webserver.cc:460] Webserver started at https://127.0.0.1:39747/ using document root <none> and password file <none>
I20260620 23:05:34.991211 12297 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:05:34.991801 12297 webserver.cc:460] Webserver started at http://127.0.0.1:33855/ using document root <none> and password file <none>
2026-06-20T23:05:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:35Z Disabled control of system clock
I20260620 23:05:35.015671 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:44863
--webserver_interface=127.12.2.126
--webserver_port=0
--builtin_ntp_servers=127.12.2.84:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:44863
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39747/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/testchainca.cert with env {}
W20260620 23:05:35.323606 17442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:35.323912 17442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:35.324044 17442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:35.331179 17442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:35.331283 17442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:35.331343 17442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:35.331429 17442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:35.331485 17442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:05:35.331574 17442 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:05:35.331622 17442 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39747/jwks.json
I20260620 23:05:35.341629 17442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:44863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:44863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39747/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:35.342921 17442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:35.344498 17442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:35.355928 17448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:35.356014 17450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:35.355959 17447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:35.356961 17442 server_base.cc:1034] running on GCE node
I20260620 23:05:35.357748 17442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:35.359207 17442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:35.360453 17442 hybrid_clock.cc:648] HybridClock initialized: now 1781996735360392 us; error 46 us; skew 500 ppm
I20260620 23:05:35.360821 17442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:05:35.366549 17442 webserver.cc:460] Webserver started at http://127.12.2.126:44307/ using document root <none> and password file <none>
I20260620 23:05:35.367306 17442 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:35.367403 17442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:35.367695 17442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:35.370822 17442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "9869cf7cfc3f428182696eeadf9f10bd"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-7g8g"
server_key: "a18bd9d1239529f9b7fb69e3f824661d"
server_key_iv: "c20b91760539361c67d40c9f14cd6761"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.371600 17442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "9869cf7cfc3f428182696eeadf9f10bd"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-7g8g"
server_key: "a18bd9d1239529f9b7fb69e3f824661d"
server_key_iv: "c20b91760539361c67d40c9f14cd6761"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.376262 17442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:05:35.379418 17456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:35.380751 17442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:05:35.380903 17442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "9869cf7cfc3f428182696eeadf9f10bd"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-7g8g"
server_key: "a18bd9d1239529f9b7fb69e3f824661d"
server_key_iv: "c20b91760539361c67d40c9f14cd6761"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.381079 17442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:35.399463 17442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:35.400278 17442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:35.400525 17442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:35.416762 17442 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:44863
I20260620 23:05:35.416774 17507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:44863 every 8 connection(s)
I20260620 23:05:35.418059 17442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:35.421454 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 17442
I20260620 23:05:35.421630 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:35.421638 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:35.422207 12297 external_mini_cluster.cc:1428] Setting key 8ba1f3fb09bf03d39dd143c9d20e4c37
I20260620 23:05:35.435221 17508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd: Bootstrap starting.
I20260620 23:05:35.437325 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:35.424007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.12.2.126:44863)
0620 23:05:35.425113 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:35.425128 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:35.425185 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:05:35.427752 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:35.427780 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:35.428391 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:35.428830 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:35.429331 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.431114 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.432390 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.433418 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.433582 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:35.434610 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:35.434621 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:35.434639 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:35.435342 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:35.435435 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:35.435459 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:35.435573 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:35.435760 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 23:05:35.436074 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":353,"threads_started":1}
I20260620 23:05:35.438771 17508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:35.439847 17508 log.cc:824] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:35.440862 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:35.442420 17508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd: No bootstrap required, opened a new log
I20260620 23:05:35.449602 17508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9869cf7cfc3f428182696eeadf9f10bd" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44863 } }
I20260620 23:05:35.449949 17508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:35.450032 17508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9869cf7cfc3f428182696eeadf9f10bd, State: Initialized, Role: FOLLOWER
I20260620 23:05:35.450533 17508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9869cf7cfc3f428182696eeadf9f10bd" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44863 } }
I20260620 23:05:35.450737 17508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:35.450851 17508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:35.451017 17508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:35.453167 17508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9869cf7cfc3f428182696eeadf9f10bd" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44863 } }
I20260620 23:05:35.453620 17508 leader_election.cc:302] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9869cf7cfc3f428182696eeadf9f10bd; no voters:
I20260620 23:05:35.454111 17508 leader_election.cc:288] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:35.454231 17513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:35.454854 17513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [term 1 LEADER]: Becoming Leader. State: Replica: 9869cf7cfc3f428182696eeadf9f10bd, State: Running, Role: LEADER
I20260620 23:05:35.455370 17513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9869cf7cfc3f428182696eeadf9f10bd" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44863 } }
I20260620 23:05:35.455927 17508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:35.457810 17514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9869cf7cfc3f428182696eeadf9f10bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9869cf7cfc3f428182696eeadf9f10bd" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44863 } } }
I20260620 23:05:35.458150 17515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9869cf7cfc3f428182696eeadf9f10bd. Latest consensus state: current_term: 1 leader_uuid: "9869cf7cfc3f428182696eeadf9f10bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9869cf7cfc3f428182696eeadf9f10bd" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44863 } } }
I20260620 23:05:35.458362 17514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:35.458488 17515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:35.459715 17521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:35.464015 17521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:35.471766 17521 catalog_manager.cc:1347] Generated new cluster ID: ee2167303faf47b5bdf03df953ff24c8
I20260620 23:05:35.471880 17521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:35.487702 17521 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:35.488929 17521 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:35.494879 17521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9869cf7cfc3f428182696eeadf9f10bd: Generated new TSK 0
I20260620 23:05:35.495792 17521 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:05:35.546470 17538 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 23:05:35.546657 17538 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:05:35.547546 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:35.519957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.12.2.126:44863)
0620 23:05:35.520120 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:35.520127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:35.520536 (+ 409us) server_negotiation.cc:407] Connection header received
0620 23:05:35.520811 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:35.520815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:35.520888 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:35.521015 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:05:35.521533 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.522256 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.523671 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.524104 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.524209 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:35.524725 (+ 516us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:05:35.546848 (+ 22123us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:05:35.547255 (+ 407us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37736: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 23:05:35.547677 17511 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37736: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 23:05:35.549163 17537 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:05:35.519879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:44863 (local address 127.0.0.1:37736)
0620 23:05:35.520441 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:35.520456 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:35.520612 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:35.521052 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:35.521058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:35.521070 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:05:35.521397 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:35.521407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.522388 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.522392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:35.523542 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:35.523553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.524282 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.524288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:35.524420 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:35.524475 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:05:35.547157 (+ 22682us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:05:35.547509 (+ 352us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.2.126:44863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":295,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.2.126:44863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39747/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 23:05:35.550596 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 17442
I20260620 23:05:35.551839 17503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:35.641589 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 17442
2026-06-20T23:05:35Z chronyd exiting
I20260620 23:05:35.687646 12297 test_util.cc:175] -----------------------------------------------
I20260620 23:05:35.687779 12297 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996696601630-12297-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (703 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 23:05:35.691429 12297 mini_oidc.cc:152] Starting JWKS server
I20260620 23:05:35.693228 12297 webserver.cc:460] Webserver started at https://127.0.0.1:43527/ using document root <none> and password file <none>
I20260620 23:05:35.693415 12297 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:05:35.694010 12297 webserver.cc:460] Webserver started at http://127.0.0.1:40449/ using document root <none> and password file <none>
2026-06-20T23:05:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:35Z Disabled control of system clock
I20260620 23:05:35.718382 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:34841
--webserver_interface=127.12.2.126
--webserver_port=0
--builtin_ntp_servers=127.12.2.84:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:34841
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43527/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/testchainca.cert with env {}
W20260620 23:05:36.022177 17545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:36.022470 17545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:36.022565 17545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:36.029803 17545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:36.029920 17545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:36.029958 17545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:36.030012 17545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:36.030053 17545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:05:36.030110 17545 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:05:36.030148 17545 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43527/jwks.json
I20260620 23:05:36.040112 17545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:34841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:34841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43527/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:36.041332 17545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:36.042584 17545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:36.053731 17550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.053777 17553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.053802 17551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:36.054565 17545 server_base.cc:1034] running on GCE node
I20260620 23:05:36.055354 17545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:36.056734 17545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:36.057948 17545 hybrid_clock.cc:648] HybridClock initialized: now 1781996736057905 us; error 37 us; skew 500 ppm
I20260620 23:05:36.058357 17545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:05:36.063499 17545 webserver.cc:460] Webserver started at http://127.12.2.126:42003/ using document root <none> and password file <none>
I20260620 23:05:36.064184 17545 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:36.064283 17545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:36.064580 17545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:36.069161 17545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "c0d60da5bb4d496484d2f9bfa2bb7c1e"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-7g8g"
server_key: "f098e01c784ac9c0c2bb1ca5743ecb8c"
server_key_iv: "c6e0bb8a68aa17a2ecfe4b603bb0141e"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.070166 17545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "c0d60da5bb4d496484d2f9bfa2bb7c1e"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-7g8g"
server_key: "f098e01c784ac9c0c2bb1ca5743ecb8c"
server_key_iv: "c6e0bb8a68aa17a2ecfe4b603bb0141e"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.076982 17545 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260620 23:05:36.083906 17559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.085721 17545 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 23:05:36.085956 17545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "c0d60da5bb4d496484d2f9bfa2bb7c1e"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-7g8g"
server_key: "f098e01c784ac9c0c2bb1ca5743ecb8c"
server_key_iv: "c6e0bb8a68aa17a2ecfe4b603bb0141e"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.086233 17545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:36.118754 17545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:36.119624 17545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:36.119859 17545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:36.136411 17545 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:34841
I20260620 23:05:36.136420 17610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:34841 every 8 connection(s)
I20260620 23:05:36.137694 17545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:36.141178 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:36.144025 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 17545
I20260620 23:05:36.144215 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:36.144956 12297 external_mini_cluster.cc:1428] Setting key dab2ca365260e3eae891368f5e14e1a6
I20260620 23:05:36.152647 17611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e: Bootstrap starting.
I20260620 23:05:36.155503 17611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:36.156525 17611 log.cc:824] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:36.157521 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.146847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.12.2.126:34841)
0620 23:05:36.147816 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.147826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.147868 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:05:36.149412 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.149432 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.149890 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.150237 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:36.150795 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.152036 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.153283 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.153939 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.154096 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.155151 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:36.155158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:36.155172 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:36.155701 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:36.155804 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:36.155830 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:36.155938 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.156134 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.156409 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":321,"threads_started":1}
I20260620 23:05:36.159494 17611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e: No bootstrap required, opened a new log
I20260620 23:05:36.161355 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:36.167627 17611 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d60da5bb4d496484d2f9bfa2bb7c1e" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34841 } }
I20260620 23:05:36.167949 17611 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:36.168031 17611 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d60da5bb4d496484d2f9bfa2bb7c1e, State: Initialized, Role: FOLLOWER
I20260620 23:05:36.168522 17611 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d60da5bb4d496484d2f9bfa2bb7c1e" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34841 } }
I20260620 23:05:36.168725 17611 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:36.168841 17611 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:36.168977 17611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:36.171123 17611 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d60da5bb4d496484d2f9bfa2bb7c1e" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34841 } }
I20260620 23:05:36.171568 17611 leader_election.cc:302] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d60da5bb4d496484d2f9bfa2bb7c1e; no voters:
I20260620 23:05:36.172056 17611 leader_election.cc:288] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:36.172180 17616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:36.172791 17616 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [term 1 LEADER]: Becoming Leader. State: Replica: c0d60da5bb4d496484d2f9bfa2bb7c1e, State: Running, Role: LEADER
I20260620 23:05:36.173269 17616 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d60da5bb4d496484d2f9bfa2bb7c1e" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34841 } }
I20260620 23:05:36.173903 17611 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:36.176007 17618 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d60da5bb4d496484d2f9bfa2bb7c1e. Latest consensus state: current_term: 1 leader_uuid: "c0d60da5bb4d496484d2f9bfa2bb7c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d60da5bb4d496484d2f9bfa2bb7c1e" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34841 } } }
I20260620 23:05:36.176127 17617 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0d60da5bb4d496484d2f9bfa2bb7c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d60da5bb4d496484d2f9bfa2bb7c1e" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34841 } } }
I20260620 23:05:36.176404 17618 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:36.176474 17617 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:36.178632 17621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:36.183176 17621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:36.190675 17621 catalog_manager.cc:1347] Generated new cluster ID: a139f8886f3742fdb5e8a4a0dba9bbff
I20260620 23:05:36.190776 17621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:36.215523 17621 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:36.216753 17621 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:36.225106 17621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0d60da5bb4d496484d2f9bfa2bb7c1e: Generated new TSK 0
I20260620 23:05:36.225888 17621 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:05:36.265376 17641 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 23:05:36.265486 17641 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:05:36.266768 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.247095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.12.2.126:34841)
0620 23:05:36.247269 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.247276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.247701 (+ 425us) server_negotiation.cc:407] Connection header received
0620 23:05:36.247966 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.247972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.248063 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.248234 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:05:36.248712 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.249818 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.251153 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.251819 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.251939 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.252317 (+ 378us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:05:36.266077 (+ 13760us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:05:36.266443 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 23:05:36.266780 17640 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:05:36.246888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34841 (local address 127.0.0.1:50708)
0620 23:05:36.247595 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.247616 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.247786 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.248258 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.248264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.248276 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:05:36.248547 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.248556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.249921 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.249924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.251029 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.251042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.251946 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.251950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.252059 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.252088 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:05:36.266302 (+ 14214us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:05:36.266487 (+ 185us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.2.126:34841: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":402,"threads_started":1}
W20260620 23:05:36.266889 17614 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 23:05:36.268606 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 17545
I20260620 23:05:36.269620 17606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:36.359995 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 17545
2026-06-20T23:05:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (719 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 23:05:36.411386 12297 mini_oidc.cc:152] Starting JWKS server
I20260620 23:05:36.413197 12297 webserver.cc:460] Webserver started at https://127.0.0.1:42519/ using document root <none> and password file <none>
I20260620 23:05:36.413416 12297 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:05:36.414016 12297 webserver.cc:460] Webserver started at http://127.0.0.1:36335/ using document root <none> and password file <none>
2026-06-20T23:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:36Z Disabled control of system clock
I20260620 23:05:36.438256 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:42945
--webserver_interface=127.12.2.126
--webserver_port=0
--builtin_ntp_servers=127.12.2.84:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:42945
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42519/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:05:36.738966 17648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:36.739255 17648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:36.739369 17648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:36.746369 17648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:36.746483 17648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:36.746548 17648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:36.746625 17648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:36.746683 17648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:05:36.746771 17648 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:05:36.746822 17648 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42519/jwks.json
I20260620 23:05:36.755659 17648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:42945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:42945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42519/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:36.756896 17648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:36.758595 17648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:36.769313 17653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.769282 17656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.769351 17654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:36.770018 17648 server_base.cc:1034] running on GCE node
I20260620 23:05:36.770741 17648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:36.772125 17648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:36.773313 17648 hybrid_clock.cc:648] HybridClock initialized: now 1781996736773264 us; error 47 us; skew 500 ppm
I20260620 23:05:36.773633 17648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:05:36.775815 17648 webserver.cc:460] Webserver started at http://127.12.2.126:41401/ using document root <none> and password file <none>
I20260620 23:05:36.776448 17648 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:36.776541 17648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:36.776839 17648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:36.779604 17648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "d8fe500dc123465286181463b553c98d"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-7g8g"
server_key: "34123af314474ef91038ec39fc201896"
server_key_iv: "973b91bbc04271243f6ec082b14e1f07"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.780354 17648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "d8fe500dc123465286181463b553c98d"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-7g8g"
server_key: "34123af314474ef91038ec39fc201896"
server_key_iv: "973b91bbc04271243f6ec082b14e1f07"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.785193 17648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:05:36.788329 17662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.789577 17648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:36.789726 17648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "d8fe500dc123465286181463b553c98d"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-7g8g"
server_key: "34123af314474ef91038ec39fc201896"
server_key_iv: "973b91bbc04271243f6ec082b14e1f07"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.789916 17648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:36.807649 17648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:36.808413 17648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:36.808656 17648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:36.825291 17648 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:42945
I20260620 23:05:36.825312 17713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:42945 every 8 connection(s)
I20260620 23:05:36.826543 17648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:36.830287 17714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:36.834353 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 17648
I20260620 23:05:36.834575 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:36.835458 12297 external_mini_cluster.cc:1428] Setting key 1e3810d93e6d64d33a12c613d60a32bc
I20260620 23:05:36.841849 17714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d: Bootstrap starting.
I20260620 23:05:36.844941 17714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:36.846192 17714 log.cc:824] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:36.848697 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.837390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.12.2.126:42945)
0620 23:05:36.838325 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.838343 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.838382 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:05:36.840447 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.840468 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.840882 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.841257 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:36.841751 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.843083 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.844319 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.844967 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.845137 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.846374 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:36.846384 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:36.846397 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:36.847069 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:36.847171 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:36.847206 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:36.847345 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.847537 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.847830 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":343,"threads_started":1}
I20260620 23:05:36.849267 17714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d: No bootstrap required, opened a new log
I20260620 23:05:36.852164 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:36.857892 17714 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe500dc123465286181463b553c98d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42945 } }
I20260620 23:05:36.858250 17714 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:36.858345 17714 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8fe500dc123465286181463b553c98d, State: Initialized, Role: FOLLOWER
I20260620 23:05:36.858861 17714 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe500dc123465286181463b553c98d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42945 } }
I20260620 23:05:36.859108 17714 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:36.859220 17714 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:36.859366 17714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:36.861784 17714 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe500dc123465286181463b553c98d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42945 } }
I20260620 23:05:36.862264 17714 leader_election.cc:302] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8fe500dc123465286181463b553c98d; no voters:
I20260620 23:05:36.862797 17714 leader_election.cc:288] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:36.862955 17719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:36.863596 17719 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [term 1 LEADER]: Becoming Leader. State: Replica: d8fe500dc123465286181463b553c98d, State: Running, Role: LEADER
I20260620 23:05:36.864097 17719 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe500dc123465286181463b553c98d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42945 } }
I20260620 23:05:36.864908 17714 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:36.866792 17721 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8fe500dc123465286181463b553c98d. Latest consensus state: current_term: 1 leader_uuid: "d8fe500dc123465286181463b553c98d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe500dc123465286181463b553c98d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42945 } } }
I20260620 23:05:36.867172 17721 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:36.867005 17720 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8fe500dc123465286181463b553c98d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fe500dc123465286181463b553c98d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42945 } } }
I20260620 23:05:36.867288 17720 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:36.876111 17726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:36.881139 17726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:36.889396 17726 catalog_manager.cc:1347] Generated new cluster ID: a65145baa3774b29b01c34ea5351d511
I20260620 23:05:36.889506 17726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:36.916782 17726 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:36.918645 17726 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:36.928134 17726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8fe500dc123465286181463b553c98d: Generated new TSK 0
I20260620 23:05:36.929165 17726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:36.957808 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.950903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.12.2.126:42945)
0620 23:05:36.951129 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.951136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.951415 (+ 279us) server_negotiation.cc:407] Connection header received
0620 23:05:36.951658 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.951663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.951734 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.951878 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:05:36.952311 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.953041 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.954370 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.954815 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.954918 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.955444 (+ 526us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:05:36.957208 (+ 1764us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42519/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:05:36.957508 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42519/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 23:05:36.958032 17743 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:05:36.950712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:42945 (local address 127.0.0.1:39348)
0620 23:05:36.951304 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.951322 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.951471 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.951909 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.951913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.951925 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:05:36.952172 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.952180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.953186 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.953190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.954227 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.954239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.955022 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.955028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.955161 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.955194 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:05:36.957406 (+ 2212us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42519/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:05:36.957627 (+ 221us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.2.126:42945: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42519/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":421,"threads_started":1}
W20260620 23:05:36.958307 17717 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42519/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260620 23:05:36.960901 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 17648
I20260620 23:05:36.961802 17709 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 23:05:36.970163 17744 webserver.cc:533] Webserver: sslize failed:
E20260620 23:05:36.970250 17744 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:05:37.051040 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 17648
2026-06-20T23:05:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (690 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T23:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:37Z Disabled control of system clock
I20260620 23:05:37.124396 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:38303
--webserver_interface=127.12.2.126
--webserver_port=0
--builtin_ntp_servers=127.12.2.84:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:38303
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/testchainca.cert with env {}
W20260620 23:05:37.425770 17749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:37.426041 17749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:37.426167 17749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:37.433265 17749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:37.433364 17749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:37.433426 17749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:37.433501 17749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:37.433568 17749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:05:37.433652 17749 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:05:37.433703 17749 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 23:05:37.443650 17749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:38303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:38303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:37.444875 17749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:37.446069 17749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:37.456722 17755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:37.456820 17754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:37.456820 17757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:37.457664 17749 server_base.cc:1034] running on GCE node
I20260620 23:05:37.458339 17749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:37.459735 17749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:37.460961 17749 hybrid_clock.cc:648] HybridClock initialized: now 1781996737460919 us; error 25 us; skew 500 ppm
I20260620 23:05:37.461321 17749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:05:37.463610 17749 webserver.cc:460] Webserver started at http://127.12.2.126:44351/ using document root <none> and password file <none>
I20260620 23:05:37.464246 17749 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:37.464334 17749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:37.464592 17749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:37.467377 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "626bfd06fee849cd851eb6c1fa3d5449"
format_stamp: "Formatted at 2026-06-20 23:05:37 on dist-test-slave-7g8g"
server_key: "4776cf1cb76f2453cb342a09d0f813de"
server_key_iv: "76b8f1cfcf7ee63572e6c773e2440069"
server_key_version: "encryptionkey@0"
I20260620 23:05:37.468084 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "626bfd06fee849cd851eb6c1fa3d5449"
format_stamp: "Formatted at 2026-06-20 23:05:37 on dist-test-slave-7g8g"
server_key: "4776cf1cb76f2453cb342a09d0f813de"
server_key_iv: "76b8f1cfcf7ee63572e6c773e2440069"
server_key_version: "encryptionkey@0"
I20260620 23:05:37.472890 17749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:05:37.476246 17763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:37.477636 17749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:05:37.477816 17749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "626bfd06fee849cd851eb6c1fa3d5449"
format_stamp: "Formatted at 2026-06-20 23:05:37 on dist-test-slave-7g8g"
server_key: "4776cf1cb76f2453cb342a09d0f813de"
server_key_iv: "76b8f1cfcf7ee63572e6c773e2440069"
server_key_version: "encryptionkey@0"
I20260620 23:05:37.478027 17749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:37.500366 17749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:37.501207 17749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:37.501479 17749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:37.517856 17749 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:38303
I20260620 23:05:37.517876 17814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:38303 every 8 connection(s)
I20260620 23:05:37.519202 17749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:37.522766 17815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:37.529497 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 17749
I20260620 23:05:37.529682 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:37.530465 12297 external_mini_cluster.cc:1428] Setting key 6d5ce5369d450e79e11e0023fad239f4
I20260620 23:05:37.533583 17815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449: Bootstrap starting.
I20260620 23:05:37.536703 17815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:37.537820 17815 log.cc:824] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:37.540746 17815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449: No bootstrap required, opened a new log
I20260620 23:05:37.543987 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:37.532422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.12.2.126:38303)
0620 23:05:37.533382 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:37.533394 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:37.533433 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:05:37.535362 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:37.535389 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:37.535865 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:37.536266 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:37.536807 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.538187 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.539777 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.540433 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.540563 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:37.541717 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:37.541725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:37.541739 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:37.542255 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:37.542340 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:37.542360 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:37.542447 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:37.542691 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 23:05:37.543022 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":328,"threads_started":1}
I20260620 23:05:37.547784 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:37.549024 17815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626bfd06fee849cd851eb6c1fa3d5449" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38303 } }
I20260620 23:05:37.549343 17815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:37.549434 17815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626bfd06fee849cd851eb6c1fa3d5449, State: Initialized, Role: FOLLOWER
I20260620 23:05:37.550040 17815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626bfd06fee849cd851eb6c1fa3d5449" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38303 } }
I20260620 23:05:37.550247 17815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:37.550372 17815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:37.550519 17815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:37.552610 17815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626bfd06fee849cd851eb6c1fa3d5449" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38303 } }
I20260620 23:05:37.553035 17815 leader_election.cc:302] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 626bfd06fee849cd851eb6c1fa3d5449; no voters:
I20260620 23:05:37.553517 17815 leader_election.cc:288] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:37.553642 17820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:37.554225 17820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [term 1 LEADER]: Becoming Leader. State: Replica: 626bfd06fee849cd851eb6c1fa3d5449, State: Running, Role: LEADER
I20260620 23:05:37.554685 17820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626bfd06fee849cd851eb6c1fa3d5449" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38303 } }
I20260620 23:05:37.555356 17815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:37.557205 17822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 626bfd06fee849cd851eb6c1fa3d5449. Latest consensus state: current_term: 1 leader_uuid: "626bfd06fee849cd851eb6c1fa3d5449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626bfd06fee849cd851eb6c1fa3d5449" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38303 } } }
I20260620 23:05:37.557188 17821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "626bfd06fee849cd851eb6c1fa3d5449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626bfd06fee849cd851eb6c1fa3d5449" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38303 } } }
I20260620 23:05:37.557507 17822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:37.557580 17821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:37.558259 17825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:37.563191 17825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:37.572074 17825 catalog_manager.cc:1347] Generated new cluster ID: 5f86582cd42b4343a4a2e2b4a7a2e7ed
I20260620 23:05:37.572194 17825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:37.597617 17825 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:37.599445 17825 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:37.608194 17825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 626bfd06fee849cd851eb6c1fa3d5449: Generated new TSK 0
I20260620 23:05:37.609174 17825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:37.700855 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:37.621285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.12.2.126:38303)
0620 23:05:37.621458 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:37.621464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:37.621927 (+ 463us) server_negotiation.cc:407] Connection header received
0620 23:05:37.622149 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:37.622154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:37.622221 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:37.622331 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:05:37.622819 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.623528 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.625475 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.625901 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.625987 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:37.626486 (+ 499us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:05:37.700261 (+ 73775us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:05:37.700592 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59964: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 23:05:37.700989 17818 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59964: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 23:05:37.701267 17844 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:05:37.621253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:38303 (local address 127.0.0.1:59964)
0620 23:05:37.621828 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:37.621848 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:37.621995 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:37.622364 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:37.622369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:37.622382 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:05:37.622678 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:37.622690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.623661 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.623667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:37.625349 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:37.625364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.626050 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.626057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:37.626236 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:37.626279 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:05:37.700559 (+ 74280us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:05:37.700869 (+ 310us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.2.126:38303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":283,"threads_started":1}
I20260620 23:05:37.702332 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 17749
I20260620 23:05:37.703734 17810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:37.801306 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 17749
2026-06-20T23:05:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (729 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (392 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (380 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:38 dist-test-slave-7g8g krb5kdc[17859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:38 dist-test-slave-7g8g krb5kdc[17859](info): set up 2 sockets
Jun 20 23:05:38 dist-test-slave-7g8g krb5kdc[17859](info): commencing operation
krb5kdc: starting...
W20260620 23:05:39.555290 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:39 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:05:39.746236 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:39381
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:39381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:40.049513 17872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:40.049793 17872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:40.049916 17872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:40.056963 17872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:40.057075 17872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:40.057156 17872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:40.057224 17872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:40.057279 17872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:40.066978 17872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:39381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:39381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:40.068277 17872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:40.069528 17872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:40.080636 17878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:40.080765 17877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:40.080797 17880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:40.081239 17872 server_base.cc:1034] running on GCE node
I20260620 23:05:40.082041 17872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:40.083452 17872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:40.084648 17872 hybrid_clock.cc:648] HybridClock initialized: now 1781996740084591 us; error 51 us; skew 500 ppm
I20260620 23:05:40.084959 17872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:40 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996740, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:40.087834 17872 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:05:40.089154 17872 webserver.cc:460] Webserver started at http://127.12.2.126:32895/ using document root <none> and password file <none>
I20260620 23:05:40.089807 17872 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:40.089921 17872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:40.090195 17872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:40.092993 17872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "0329677bfe664c269669df6551b17437"
format_stamp: "Formatted at 2026-06-20 23:05:40 on dist-test-slave-7g8g"
server_key: "54dacf82869de74543c8ed53569b9af7"
server_key_iv: "f853da0ff618b7131e8ff316ee201999"
server_key_version: "encryptionkey@0"
I20260620 23:05:40.093703 17872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "0329677bfe664c269669df6551b17437"
format_stamp: "Formatted at 2026-06-20 23:05:40 on dist-test-slave-7g8g"
server_key: "54dacf82869de74543c8ed53569b9af7"
server_key_iv: "f853da0ff618b7131e8ff316ee201999"
server_key_version: "encryptionkey@0"
I20260620 23:05:40.098546 17872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:05:40.102176 17887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:40.103507 17872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:40.103659 17872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "0329677bfe664c269669df6551b17437"
format_stamp: "Formatted at 2026-06-20 23:05:40 on dist-test-slave-7g8g"
server_key: "54dacf82869de74543c8ed53569b9af7"
server_key_iv: "f853da0ff618b7131e8ff316ee201999"
server_key_version: "encryptionkey@0"
I20260620 23:05:40.103825 17872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:40.126050 17872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:40.130045 17872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:40.130306 17872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:40.146448 17872 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:39381
I20260620 23:05:40.146481 17938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:39381 every 8 connection(s)
I20260620 23:05:40.147768 17872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:40.151353 17939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:40.152908 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 17872
I20260620 23:05:40.153077 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:40.153666 12297 external_mini_cluster.cc:1428] Setting key 7ef0e5a8acb7cd6f69e2c7797cb1b0dd
I20260620 23:05:40.163687 17939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437: Bootstrap starting.
I20260620 23:05:40.167203 17939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437: Neither blocks nor log segments found. Creating new log.
Jun 20 23:05:40 dist-test-slave-7g8g krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:40.168215 17939 log.cc:824] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:40.171041 17939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437: No bootstrap required, opened a new log
I20260620 23:05:40.176527 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:40.155567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.12.2.126:39381)
0620 23:05:40.156317 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:40.156332 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:40.156381 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:05:40.158671 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:40.158698 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:40.159211 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:40.159635 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:40.160791 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:40.162771 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:40.163978 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:40.164934 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:40.165107 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:40.168967 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:40.168975 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:40.168992 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:40.169028 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:40.171860 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:40.172620 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:40.172629 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:40.172640 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:40.172856 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:40.173371 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:40.173400 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:40.173403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:40.174100 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:40.174392 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:40.175117 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 23:05:40.175570 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":280,"threads_started":1}
I20260620 23:05:40.180181 17939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0329677bfe664c269669df6551b17437" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39381 } }
I20260620 23:05:40.180505 17939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:40.180591 17939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0329677bfe664c269669df6551b17437, State: Initialized, Role: FOLLOWER
I20260620 23:05:40.181188 17939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0329677bfe664c269669df6551b17437" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39381 } }
I20260620 23:05:40.181448 17939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:40.181584 17939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:40.181748 17939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:40.183768 17939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0329677bfe664c269669df6551b17437" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39381 } }
I20260620 23:05:40.184190 17939 leader_election.cc:302] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0329677bfe664c269669df6551b17437; no voters:
I20260620 23:05:40.184651 17939 leader_election.cc:288] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:40.184767 17944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:40.185338 17944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [term 1 LEADER]: Becoming Leader. State: Replica: 0329677bfe664c269669df6551b17437, State: Running, Role: LEADER
I20260620 23:05:40.185827 17944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0329677bfe664c269669df6551b17437" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39381 } }
I20260620 23:05:40.186437 17939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:40.188323 17945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0329677bfe664c269669df6551b17437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0329677bfe664c269669df6551b17437" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39381 } } }
I20260620 23:05:40.188678 17945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:40.193909 17946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0329677bfe664c269669df6551b17437. Latest consensus state: current_term: 1 leader_uuid: "0329677bfe664c269669df6551b17437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0329677bfe664c269669df6551b17437" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39381 } } }
I20260620 23:05:40.194187 17946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437 [sys.catalog]: This master's current role is: LEADER
W20260620 23:05:40.197836 17960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:05:40.197932 17960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:05:40.198010 17949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:40.199548 17949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:40.207522 17949 catalog_manager.cc:1347] Generated new cluster ID: b8aa1adc6a2344b0b7bd2b13bb9cd7ab
I20260620 23:05:40.207633 17949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:40.246783 17949 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:40.248561 17949 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:40.264318 17949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0329677bfe664c269669df6551b17437: Generated new TSK 0
I20260620 23:05:40.265345 17949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:05:40.319502 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39381
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:40.622454 17965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:40.622723 17965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:40.622848 17965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:40.629981 17965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:40.630080 17965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:40.630220 17965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:40.639690 17965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:40.640916 17965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:40.642150 17965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:40.654263 17970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:40.654314 17971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:40.654367 17973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:40.655400 17965 server_base.cc:1034] running on GCE node
I20260620 23:05:40.655961 17965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:40.656833 17965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:40.658027 17965 hybrid_clock.cc:648] HybridClock initialized: now 1781996740657975 us; error 52 us; skew 500 ppm
I20260620 23:05:40.658322 17965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:40 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996740, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:40.664569 17965 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:05:40.666009 17965 webserver.cc:460] Webserver started at http://127.12.2.65:35159/ using document root <none> and password file <none>
I20260620 23:05:40.666659 17965 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:40.666771 17965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:40.667111 17965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:40.669889 17965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "bd3e2f2c0905465bb93442dbbd7169fd"
format_stamp: "Formatted at 2026-06-20 23:05:40 on dist-test-slave-7g8g"
server_key: "dc5989b5af8ed33be6c446fff08bdc85"
server_key_iv: "5a8d7d510038f5148a07e53eae18938e"
server_key_version: "encryptionkey@0"
I20260620 23:05:40.670620 17965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "bd3e2f2c0905465bb93442dbbd7169fd"
format_stamp: "Formatted at 2026-06-20 23:05:40 on dist-test-slave-7g8g"
server_key: "dc5989b5af8ed33be6c446fff08bdc85"
server_key_iv: "5a8d7d510038f5148a07e53eae18938e"
server_key_version: "encryptionkey@0"
I20260620 23:05:40.675729 17965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:05:40.679306 17980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:40.680583 17965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:05:40.680709 17965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "bd3e2f2c0905465bb93442dbbd7169fd"
format_stamp: "Formatted at 2026-06-20 23:05:40 on dist-test-slave-7g8g"
server_key: "dc5989b5af8ed33be6c446fff08bdc85"
server_key_iv: "5a8d7d510038f5148a07e53eae18938e"
server_key_version: "encryptionkey@0"
I20260620 23:05:40.680873 17965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:40.718892 17965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:40.722633 17965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:40.722857 17965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:40.723742 17965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:40.724936 17965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:40.725011 17965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:40.725095 17965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:40.725136 17965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:40.754314 17965 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:42873
I20260620 23:05:40.754386 18092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:42873 every 8 connection(s)
I20260620 23:05:40.755554 17965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:40.757396 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 17965
I20260620 23:05:40.757606 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:40.758306 12297 external_mini_cluster.cc:1428] Setting key f673a39f85a4f911ccee6cd5daa1f6af
Jun 20 23:05:40 dist-test-slave-7g8g krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996740, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:40.777352 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:40.758332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:44937 (local address 127.12.2.126:39381)
0620 23:05:40.758761 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:40.758767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:40.759852 (+ 1085us) server_negotiation.cc:407] Connection header received
0620 23:05:40.762374 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:40.762378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:40.762437 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:40.762524 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:40.764517 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:40.765215 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:40.767439 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:40.768047 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:40.768195 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:40.771443 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:40.771449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:40.771453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:40.771489 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:40.774242 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:40.774905 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:40.774910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:40.774913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:40.775087 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:40.775559 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:40.775564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:40.775567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:40.775909 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:40.776062 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:40.776748 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 23:05:40.776972 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":213,"threads_started":1}
I20260620 23:05:40.778213 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:40.758873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:39381 (local address 127.12.2.65:44937)
0620 23:05:40.759698 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:40.759742 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:40.762053 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:40.762789 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:40.762801 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:40.763520 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:40.764330 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:40.764357 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:40.765370 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:40.765375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:40.767261 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:40.767279 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:40.768287 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:40.768292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:40.768491 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:40.769107 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:40.769130 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:40.771211 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:40.774369 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:40.774375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:40.774388 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:40.774786 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:40.775242 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:40.775246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:40.775248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:40.775440 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:40.776030 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:40.776036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:40.776375 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 23:05:40.776620 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 23:05:40.777014 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":341,"threads_started":1}
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
I20260620 23:05:40.779678 18093 heartbeater.cc:347] Connected to a master server at 127.12.2.126:39381
I20260620 23:05:40.779980 18093 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:40.780699 18093 heartbeater.cc:511] Master 127.12.2.126:39381 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
I20260620 23:05:40.783025 17904 ts_manager.cc:194] Registered new tserver with Master: bd3e2f2c0905465bb93442dbbd7169fd (127.12.2.65:42873)
I20260620 23:05:40.784941 17904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:44937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:05:40.803963 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39381
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:41.108208 18099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:41.108491 18099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:41.108619 18099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:41.115748 18099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:41.115872 18099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:41.116020 18099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:41.125670 18099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:41.126894 18099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:41.128288 18099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:41.140483 18104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:41.140378 18105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:41.140408 18107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:41.141211 18099 server_base.cc:1034] running on GCE node
I20260620 23:05:41.141786 18099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:41.142738 18099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:41.143954 18099 hybrid_clock.cc:648] HybridClock initialized: now 1781996741143900 us; error 49 us; skew 500 ppm
I20260620 23:05:41.144273 18099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:41.147565 18099 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:05:41.149348 18099 webserver.cc:460] Webserver started at http://127.12.2.66:40519/ using document root <none> and password file <none>
I20260620 23:05:41.150187 18099 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:41.150327 18099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:41.150681 18099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:41.154690 18099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "4515c96deda84a68919ceac9c83138c2"
format_stamp: "Formatted at 2026-06-20 23:05:41 on dist-test-slave-7g8g"
server_key: "42fac8c59c2b5ac029d538508225ea2e"
server_key_iv: "65d26e97db99b95a426a0c185b1b616a"
server_key_version: "encryptionkey@0"
I20260620 23:05:41.155738 18099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "4515c96deda84a68919ceac9c83138c2"
format_stamp: "Formatted at 2026-06-20 23:05:41 on dist-test-slave-7g8g"
server_key: "42fac8c59c2b5ac029d538508225ea2e"
server_key_iv: "65d26e97db99b95a426a0c185b1b616a"
server_key_version: "encryptionkey@0"
I20260620 23:05:41.162559 18099 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 23:05:41.167104 18114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:41.168458 18099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:41.168591 18099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "4515c96deda84a68919ceac9c83138c2"
format_stamp: "Formatted at 2026-06-20 23:05:41 on dist-test-slave-7g8g"
server_key: "42fac8c59c2b5ac029d538508225ea2e"
server_key_iv: "65d26e97db99b95a426a0c185b1b616a"
server_key_version: "encryptionkey@0"
I20260620 23:05:41.168761 18099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:41.185190 18099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:41.189076 18099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:41.189332 18099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:41.190347 18099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:41.191671 18099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:41.191740 18099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:41.191830 18099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:41.191871 18099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:41.220271 18099 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:33051
I20260620 23:05:41.220297 18226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:33051 every 8 connection(s)
I20260620 23:05:41.221459 18099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:41.230610 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 18099
I20260620 23:05:41.230788 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:41.231253 12297 external_mini_cluster.cc:1428] Setting key 68d0e2efb60170ea03ff127aa80fc004
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:41.239965 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:41.223837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:42147 (local address 127.12.2.126:39381)
0620 23:05:41.223994 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.223998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.225132 (+ 1134us) server_negotiation.cc:407] Connection header received
0620 23:05:41.226832 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.226837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.226916 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.227033 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.228465 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.229160 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.230850 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.231303 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.231421 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:41.234790 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:41.234795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:41.234798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:41.234827 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:41.237219 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.237884 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.237889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.237893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.238000 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.238384 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.238389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.238392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.238642 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:41.238766 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:41.239452 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 23:05:41.239655 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:05:41.240943 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:41.224268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:39381 (local address 127.12.2.66:42147)
0620 23:05:41.224981 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:41.225017 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:41.226574 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:41.227264 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:41.227273 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:41.227750 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:41.228302 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:41.228318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.229307 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.229311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:41.230684 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:41.230699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.231524 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.231531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:41.231822 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:41.232418 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:41.232440 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:41.234586 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:41.237353 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:41.237362 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:41.237378 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:41.237779 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:41.238105 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:41.238110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:41.238114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:41.238285 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:41.238765 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:41.238773 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:41.239127 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 23:05:41.239361 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:05:41.239869 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":279,"threads_started":1}
I20260620 23:05:41.242280 18227 heartbeater.cc:347] Connected to a master server at 127.12.2.126:39381
I20260620 23:05:41.242547 18227 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:41.243249 18227 heartbeater.cc:511] Master 127.12.2.126:39381 requested a full tablet report, sending...
I20260620 23:05:41.244621 17904 ts_manager.cc:194] Registered new tserver with Master: 4515c96deda84a68919ceac9c83138c2 (127.12.2.66:33051)
I20260620 23:05:41.245714 17904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:42147
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:05:41.271363 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39381
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:41.580099 18232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:41.580364 18232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:41.580478 18232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:41.588001 18232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:41.588096 18232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:41.588246 18232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:41.597644 18232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:41.598845 18232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:41.600078 18232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:41.612390 18237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:41.612447 18238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:41.612457 18240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:41.613045 18232 server_base.cc:1034] running on GCE node
I20260620 23:05:41.613550 18232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:41.614392 18232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:41.615571 18232 hybrid_clock.cc:648] HybridClock initialized: now 1781996741615522 us; error 44 us; skew 500 ppm
I20260620 23:05:41.615870 18232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:41.618835 18232 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:05:41.620108 18232 webserver.cc:460] Webserver started at http://127.12.2.67:32997/ using document root <none> and password file <none>
I20260620 23:05:41.620710 18232 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:41.620802 18232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:41.621045 18232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:41.623775 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "3266bab6592f4f50bfc0f0d96adc69d3"
format_stamp: "Formatted at 2026-06-20 23:05:41 on dist-test-slave-7g8g"
server_key: "967c903730443832ff0ea37a268d24b0"
server_key_iv: "95295c74e7f3ecd1454f8ddd3be210bf"
server_key_version: "encryptionkey@0"
I20260620 23:05:41.624456 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "3266bab6592f4f50bfc0f0d96adc69d3"
format_stamp: "Formatted at 2026-06-20 23:05:41 on dist-test-slave-7g8g"
server_key: "967c903730443832ff0ea37a268d24b0"
server_key_iv: "95295c74e7f3ecd1454f8ddd3be210bf"
server_key_version: "encryptionkey@0"
I20260620 23:05:41.629132 18232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:05:41.632452 18247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:41.633760 18232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:05:41.633894 18232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "3266bab6592f4f50bfc0f0d96adc69d3"
format_stamp: "Formatted at 2026-06-20 23:05:41 on dist-test-slave-7g8g"
server_key: "967c903730443832ff0ea37a268d24b0"
server_key_iv: "95295c74e7f3ecd1454f8ddd3be210bf"
server_key_version: "encryptionkey@0"
I20260620 23:05:41.634057 18232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:41.656486 18232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:41.660586 18232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:41.660827 18232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:41.661705 18232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:41.663007 18232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:41.663079 18232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:41.663159 18232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:41.663198 18232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:41.691129 18232 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:34099
I20260620 23:05:41.691176 18359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:34099 every 8 connection(s)
I20260620 23:05:41.692323 18232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:41.693502 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 18232
I20260620 23:05:41.693662 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:41.694062 12297 external_mini_cluster.cc:1428] Setting key bc56ba1d1a6e1218d52489500ca70e9a
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:41.711163 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:41.695970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:58029 (local address 127.12.2.126:39381)
0620 23:05:41.696163 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.696168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.697359 (+ 1191us) server_negotiation.cc:407] Connection header received
0620 23:05:41.699051 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.699055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.699118 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.699197 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.700520 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.701187 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.702580 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.703025 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.703114 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:41.706149 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:41.706154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:41.706157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:41.706187 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:41.708850 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.709457 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.709460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.709462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.709560 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.709854 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.709856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.709858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.710085 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:41.710195 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:41.710736 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 23:05:41.710892 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:05:41.712049 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:41.696493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:39381 (local address 127.12.2.67:58029)
0620 23:05:41.697211 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:41.697247 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:41.698806 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:41.699373 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:41.699381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:41.699851 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:41.700399 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:41.700415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.701301 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.701304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:41.702480 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:41.702491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.703196 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.703201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:41.703384 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:41.703996 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:41.704017 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:41.705979 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:41.708957 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:41.708963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:41.708974 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:41.709377 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:41.709621 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:41.709624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:41.709626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:41.709783 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:41.710176 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:41.710181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:41.710461 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 23:05:41.710684 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 23:05:41.711073 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":298,"threads_started":1}
I20260620 23:05:41.713196 18360 heartbeater.cc:347] Connected to a master server at 127.12.2.126:39381
I20260620 23:05:41.713466 18360 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:41.714088 18360 heartbeater.cc:511] Master 127.12.2.126:39381 requested a full tablet report, sending...
I20260620 23:05:41.715366 17904 ts_manager.cc:194] Registered new tserver with Master: 3266bab6592f4f50bfc0f0d96adc69d3 (127.12.2.67:34099)
I20260620 23:05:41.716087 17904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:58029
I20260620 23:05:41.720984 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/test-admin.keytab.
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:41.743743 12297 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 23:05:41.748234 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:05:41.746041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.12.2.65:42873)
0620 23:05:41.746540 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.746547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.746575 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:05:41.746779 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.746787 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.747036 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.747173 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.747959 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39046: BlockingRecv error: recv got EOF from 127.0.0.1:39046 (error 108)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":253,"threads_started":1}
W20260620 23:05:41.748364 12297 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:41.749781 12297 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:41.751082 12297 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:05:41.765512 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:41.752534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.12.2.65:42873)
0620 23:05:41.752709 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.752715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.752841 (+ 126us) server_negotiation.cc:407] Connection header received
0620 23:05:41.752968 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.752972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.753028 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.753101 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.754049 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.755547 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.756765 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.757574 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.757703 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:41.760874 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:41.760884 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:41.760896 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:41.760927 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:41.763384 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.763981 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.763986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.763989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.764099 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.764362 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.764365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.764366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.764658 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:41.764867 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:41.765103 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 23:05:41.765279 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 23:05:41.770769 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:05:41.769120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.12.2.65:42873)
0620 23:05:41.769335 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.769341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.769567 (+ 226us) server_negotiation.cc:407] Connection header received
0620 23:05:41.769751 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.769757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.769816 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.769889 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.770585 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39066: BlockingRecv error: recv got EOF from 127.0.0.1:39066 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 23:05:41.771006 12297 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:41.772322 12297 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:41.773518 12297 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:05:41.785938 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:41.774604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.12.2.65:42873)
0620 23:05:41.774757 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.774762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.775066 (+ 304us) server_negotiation.cc:407] Connection header received
0620 23:05:41.775205 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.775210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.775264 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.775333 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.776196 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.776853 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.778029 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.778438 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.778539 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:41.781577 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:41.781582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:41.781585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:41.781612 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:41.783733 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.784322 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.784328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.784331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.784438 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.784760 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.784765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.784769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.785032 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:41.785251 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:41.785517 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:05:41.785699 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:05:41.787901 18093 heartbeater.cc:503] Master 127.12.2.126:39381 was elected leader, sending a full tablet report...
W20260620 23:05:41.789932 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:05:41.788165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.12.2.65:42873)
0620 23:05:41.788297 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.788302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.788670 (+ 368us) server_negotiation.cc:407] Connection header received
0620 23:05:41.788843 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.788846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.788903 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.788972 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.789729 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39084: BlockingRecv error: recv got EOF from 127.0.0.1:39084 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 23:05:41.790076 12297 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996696601630-12297-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:41.791412 12297 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:41.792599 12297 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:05:41.805240 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:41.793861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.12.2.65:42873)
0620 23:05:41.794037 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.794042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.794303 (+ 261us) server_negotiation.cc:407] Connection header received
0620 23:05:41.794477 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.794481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.794534 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.794613 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.795520 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.796200 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.797314 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.797713 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.797795 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:41.800849 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:41.800854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:41.800857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:41.800938 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:41.803176 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.803729 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.803733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.803735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.803852 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.804107 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.804110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.804112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.804352 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:41.804571 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:41.804733 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:05:41.804883 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:05:41.806772 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 17965
I20260620 23:05:41.817615 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:41.807451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.12.2.65:42873)
0620 23:05:41.807591 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.807596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.808079 (+ 483us) server_negotiation.cc:407] Connection header received
0620 23:05:41.808197 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.808200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.808250 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.808324 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.809084 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.809747 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.811038 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.811419 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.811504 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:41.813427 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:41.813431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:41.813434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:41.813459 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:41.815656 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.816189 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.816192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.816194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.816297 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.816633 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.816636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.816638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.816843 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:41.817015 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:41.817218 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 23:05:41.817379 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:05:41.818101 18088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:41.898697 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 17965
I20260620 23:05:41.911060 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 18099
Jun 20 23:05:41 dist-test-slave-7g8g krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:05:41.925539 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:41.911936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.12.2.66:33051)
0620 23:05:41.912457 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.912463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.912487 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:05:41.912549 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.912555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.912763 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.912891 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.913757 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.914808 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.916244 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:41.916771 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:41.916861 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:41.920290 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:41.920301 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:41.920315 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:41.920347 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:41.922918 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.923526 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.923536 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.923540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.923685 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:41.924045 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:41.924050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:41.924053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:41.924414 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:41.924698 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:41.925010 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 23:05:41.925243 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":284,"threads_started":1}
I20260620 23:05:41.926709 18222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:42.005874 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 18099
I20260620 23:05:42.017817 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 18232
Jun 20 23:05:42 dist-test-slave-7g8g krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:05:42.032526 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:42.018714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.12.2.67:34099)
0620 23:05:42.019350 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:42.019357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:42.019384 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:05:42.019441 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:42.019447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:42.019661 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:42.019811 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:42.020851 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:42.021822 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:42.023119 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:42.023670 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:42.023779 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:42.027451 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:42.027461 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:42.027476 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:42.027514 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:42.029911 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:42.030533 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:42.030540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:42.030543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:42.030661 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:42.031103 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:42.031108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:42.031111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:42.031443 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:42.031729 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:42.031993 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 23:05:42.032252 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":338,"threads_started":1}
I20260620 23:05:42.033780 18355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:42.114615 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 18232
I20260620 23:05:42.126729 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 17872
I20260620 23:05:42.127852 17934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:42.228812 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 17872
2026-06-20T23:05:42Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3659 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:42 dist-test-slave-7g8g krb5kdc[18387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:42 dist-test-slave-7g8g krb5kdc[18387](info): set up 2 sockets
Jun 20 23:05:42 dist-test-slave-7g8g krb5kdc[18387](info): commencing operation
krb5kdc: starting...
W20260620 23:05:43.219314 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:43 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.126.keytab.
Entry for principal oryx/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.126.keytab.
I20260620 23:05:43.416503 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:41439
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.126.keytab
--principal=oryx/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:43.718240 18400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:43.718508 18400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:43.718602 18400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:43.725620 18400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:43.725728 18400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:43.725771 18400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:43.725832 18400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:43.725874 18400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:43.735312 18400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:41439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.126.keytab
--principal=oryx/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:43.736518 18400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:43.737744 18400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:43.748530 18408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:43.748574 18406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:43.748607 18405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:43.749439 18400 server_base.cc:1034] running on GCE node
I20260620 23:05:43.750140 18400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:43.751485 18400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:43.752671 18400 hybrid_clock.cc:648] HybridClock initialized: now 1781996743752629 us; error 37 us; skew 500 ppm
I20260620 23:05:43.752995 18400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:43 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:43.755859 18400 init.cc:377] Logged in from keytab as oryx/127.12.2.126@KRBTEST.COM (short username oryx)
I20260620 23:05:43.757305 18400 webserver.cc:460] Webserver started at http://127.12.2.126:43885/ using document root <none> and password file <none>
I20260620 23:05:43.757977 18400 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:43.758090 18400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:43.758364 18400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:43.761235 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "07a99eb094dc4ae295ad268a70895388"
format_stamp: "Formatted at 2026-06-20 23:05:43 on dist-test-slave-7g8g"
server_key: "bedda9b727a5d1cffe3b49844ea4f15c"
server_key_iv: "99c3aafb936dea42aa510175b0dcb349"
server_key_version: "encryptionkey@0"
I20260620 23:05:43.761967 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "07a99eb094dc4ae295ad268a70895388"
format_stamp: "Formatted at 2026-06-20 23:05:43 on dist-test-slave-7g8g"
server_key: "bedda9b727a5d1cffe3b49844ea4f15c"
server_key_iv: "99c3aafb936dea42aa510175b0dcb349"
server_key_version: "encryptionkey@0"
I20260620 23:05:43.766855 18400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:05:43.770028 18415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:43.771401 18400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:05:43.771551 18400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "07a99eb094dc4ae295ad268a70895388"
format_stamp: "Formatted at 2026-06-20 23:05:43 on dist-test-slave-7g8g"
server_key: "bedda9b727a5d1cffe3b49844ea4f15c"
server_key_iv: "99c3aafb936dea42aa510175b0dcb349"
server_key_version: "encryptionkey@0"
I20260620 23:05:43.771723 18400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:43.791124 18400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:43.794963 18400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:43.795202 18400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:43.811728 18400 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:41439
I20260620 23:05:43.811789 18466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:41439 every 8 connection(s)
I20260620 23:05:43.813014 18400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:43.814263 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 18400
I20260620 23:05:43.814473 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:43.815442 12297 external_mini_cluster.cc:1428] Setting key 94f7839d0d8ffbe5d41163ae648edb76
I20260620 23:05:43.817688 18467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:05:43 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:43.830665 18467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388: Bootstrap starting.
I20260620 23:05:43.833855 18467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:43.834916 18467 log.cc:824] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:43.837602 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:43.817491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.12.2.126:41439)
0620 23:05:43.818194 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:43.818205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:43.818245 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:05:43.819797 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:43.819818 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:43.820192 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:43.820516 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:43.822338 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:43.823707 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:43.825471 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:43.826096 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:43.826220 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:43.830833 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:43.830843 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:43.830863 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:43.830907 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:43.833608 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:43.834293 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:43.834303 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:43.834309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:43.834472 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:43.834823 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:43.834828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:43.834830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:43.835383 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:43.835701 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:43.836125 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 23:05:43.836638 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":316,"threads_started":1}
I20260620 23:05:43.837970 18467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388: No bootstrap required, opened a new log
I20260620 23:05:43.845110 18467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } }
I20260620 23:05:43.845448 18467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:43.845533 18467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a99eb094dc4ae295ad268a70895388, State: Initialized, Role: FOLLOWER
I20260620 23:05:43.846120 18467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } }
I20260620 23:05:43.846366 18467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:43.846493 18467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:43.846652 18467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:43.848719 18467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } }
I20260620 23:05:43.849195 18467 leader_election.cc:302] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07a99eb094dc4ae295ad268a70895388; no voters:
I20260620 23:05:43.849720 18467 leader_election.cc:288] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:43.849822 18472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:43.850399 18472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 1 LEADER]: Becoming Leader. State: Replica: 07a99eb094dc4ae295ad268a70895388, State: Running, Role: LEADER
I20260620 23:05:43.850898 18472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } }
I20260620 23:05:43.851614 18467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:43.853952 18473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } }
I20260620 23:05:43.854205 18473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:43.855947 18479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:43.855795 18474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a99eb094dc4ae295ad268a70895388. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } }
I20260620 23:05:43.856077 18474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:43.860293 18479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:43.869798 18479 catalog_manager.cc:1347] Generated new cluster ID: 1befd225d77342798bc52be4cecf56e2
I20260620 23:05:43.869927 18479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:43.884547 18479 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:43.885855 18479 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:43.895171 18479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388: Generated new TSK 0
I20260620 23:05:43.895979 18479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab.
Entry for principal oryx/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab.
I20260620 23:05:43.947978 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:41439
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab
--principal=oryx/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:44.266157 18493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:44.266443 18493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:44.266561 18493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:44.273866 18493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:44.273988 18493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:44.274147 18493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:44.284102 18493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab
--principal=oryx/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:44.285413 18493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:44.286746 18493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:44.299834 18499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:44.299860 18498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:44.299834 18501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:44.300552 18493 server_base.cc:1034] running on GCE node
I20260620 23:05:44.301128 18493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:44.302066 18493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:44.303269 18493 hybrid_clock.cc:648] HybridClock initialized: now 1781996744303223 us; error 50 us; skew 500 ppm
I20260620 23:05:44.303567 18493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:44 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996744, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:44.306581 18493 init.cc:377] Logged in from keytab as oryx/127.12.2.65@KRBTEST.COM (short username oryx)
I20260620 23:05:44.308015 18493 webserver.cc:460] Webserver started at http://127.12.2.65:40137/ using document root <none> and password file <none>
I20260620 23:05:44.308667 18493 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:44.308766 18493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:44.309011 18493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:44.311868 18493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "c5bae359efc64d17b9591adc8dc7b185"
format_stamp: "Formatted at 2026-06-20 23:05:44 on dist-test-slave-7g8g"
server_key: "57fc910766059ec6f6b6a24da23cceeb"
server_key_iv: "067db55f5bd6178372149e963ca00b19"
server_key_version: "encryptionkey@0"
I20260620 23:05:44.312582 18493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "c5bae359efc64d17b9591adc8dc7b185"
format_stamp: "Formatted at 2026-06-20 23:05:44 on dist-test-slave-7g8g"
server_key: "57fc910766059ec6f6b6a24da23cceeb"
server_key_iv: "067db55f5bd6178372149e963ca00b19"
server_key_version: "encryptionkey@0"
I20260620 23:05:44.317811 18493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:05:44.321156 18508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:44.322458 18493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:44.322589 18493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "c5bae359efc64d17b9591adc8dc7b185"
format_stamp: "Formatted at 2026-06-20 23:05:44 on dist-test-slave-7g8g"
server_key: "57fc910766059ec6f6b6a24da23cceeb"
server_key_iv: "067db55f5bd6178372149e963ca00b19"
server_key_version: "encryptionkey@0"
I20260620 23:05:44.322748 18493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:44.340276 18493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:44.344132 18493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:44.344419 18493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:44.345350 18493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:44.346633 18493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:44.346704 18493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:44.346788 18493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:44.346832 18493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:44.376929 18493 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:39893
I20260620 23:05:44.376997 18620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:39893 every 8 connection(s)
I20260620 23:05:44.378160 18493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:44.386790 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 18493
I20260620 23:05:44.387008 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:44.387662 12297 external_mini_cluster.cc:1428] Setting key 7dd6bb2d4c2fb4ecdc9c88678816e4c1
Jun 20 23:05:44 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996744, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:44.398545 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:44.380684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:36495 (local address 127.12.2.126:41439)
0620 23:05:44.381128 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:44.381133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:44.382102 (+ 969us) server_negotiation.cc:407] Connection header received
0620 23:05:44.384013 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:44.384020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:44.384105 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:44.384209 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:44.385734 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.386613 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.388306 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.388847 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.388978 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:44.392875 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:44.392880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:44.392884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:44.392916 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:44.395511 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:44.396273 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:44.396279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:44.396283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:44.396414 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:44.396825 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:44.396829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:44.396832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:44.397206 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:44.397331 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:44.397996 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 23:05:44.398209 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":211,"threads_started":1}
I20260620 23:05:44.399374 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:44.381249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.65:36495)
0620 23:05:44.381942 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:44.381988 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:44.383735 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:44.384460 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:44.384469 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:44.384981 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:44.385559 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:44.385577 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.386781 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.386786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:44.388165 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:44.388177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.389044 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.389048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:44.389245 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:44.389965 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:44.390000 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:44.392638 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:44.395642 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:44.395650 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:44.395667 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:44.396146 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:44.396504 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:44.396507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:44.396509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:44.396714 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:44.397313 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:44.397319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:44.397668 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 23:05:44.397888 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 23:05:44.398250 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":275,"threads_started":1}
I20260620 23:05:44.400831 18621 heartbeater.cc:347] Connected to a master server at 127.12.2.126:41439
I20260620 23:05:44.401149 18621 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:44.401959 18621 heartbeater.cc:511] Master 127.12.2.126:41439 requested a full tablet report, sending...
I20260620 23:05:44.404054 18432 ts_manager.cc:194] Registered new tserver with Master: c5bae359efc64d17b9591adc8dc7b185 (127.12.2.65:39893)
I20260620 23:05:44.405501 18432 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:36495
WARNING: no policy specified for oryx/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab.
Entry for principal oryx/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab.
I20260620 23:05:44.432025 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:41439
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab
--principal=oryx/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:44.741732 18627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:44.742027 18627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:44.742164 18627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:44.749246 18627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:44.749368 18627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:44.749524 18627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:44.759184 18627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab
--principal=oryx/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:44.760437 18627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:44.761705 18627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:44.774034 18633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:44.774065 18632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:44.774150 18635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:44.774705 18627 server_base.cc:1034] running on GCE node
I20260620 23:05:44.775399 18627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:44.776299 18627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:44.777509 18627 hybrid_clock.cc:648] HybridClock initialized: now 1781996744777437 us; error 51 us; skew 500 ppm
I20260620 23:05:44.777854 18627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:44 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996744, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:44.780835 18627 init.cc:377] Logged in from keytab as oryx/127.12.2.66@KRBTEST.COM (short username oryx)
I20260620 23:05:44.782217 18627 webserver.cc:460] Webserver started at http://127.12.2.66:37235/ using document root <none> and password file <none>
I20260620 23:05:44.782842 18627 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:44.782971 18627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:44.783252 18627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:44.785933 18627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "174c8d98e2bc45fcb171db3de048efbf"
format_stamp: "Formatted at 2026-06-20 23:05:44 on dist-test-slave-7g8g"
server_key: "4e171696d5b6490acb6a1f5293cb3552"
server_key_iv: "3f55c496fbeb06a552a70e0766c4b5af"
server_key_version: "encryptionkey@0"
I20260620 23:05:44.786633 18627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "174c8d98e2bc45fcb171db3de048efbf"
format_stamp: "Formatted at 2026-06-20 23:05:44 on dist-test-slave-7g8g"
server_key: "4e171696d5b6490acb6a1f5293cb3552"
server_key_iv: "3f55c496fbeb06a552a70e0766c4b5af"
server_key_version: "encryptionkey@0"
I20260620 23:05:44.791347 18627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260620 23:05:44.794516 18642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:44.795845 18627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:44.795994 18627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "174c8d98e2bc45fcb171db3de048efbf"
format_stamp: "Formatted at 2026-06-20 23:05:44 on dist-test-slave-7g8g"
server_key: "4e171696d5b6490acb6a1f5293cb3552"
server_key_iv: "3f55c496fbeb06a552a70e0766c4b5af"
server_key_version: "encryptionkey@0"
I20260620 23:05:44.796164 18627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:44.808550 18627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:44.812449 18627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:44.812743 18627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:44.813767 18627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:44.815312 18627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:44.815395 18627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:44.815508 18627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:44.815564 18627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:44.848659 18627 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:46755
I20260620 23:05:44.848699 18754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:46755 every 8 connection(s)
I20260620 23:05:44.849858 18627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:44.851272 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 18627
I20260620 23:05:44.851466 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:44.851912 12297 external_mini_cluster.cc:1428] Setting key 643d3cbcff9c6320e1403578b9e11f78
Jun 20 23:05:44 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996744, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.12.2.67@KRBTEST.COM; defaulting to no policy
I20260620 23:05:44.869212 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:44.852401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:52699 (local address 127.12.2.126:41439)
0620 23:05:44.852564 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:44.852568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:44.853870 (+ 1302us) server_negotiation.cc:407] Connection header received
0620 23:05:44.855721 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:44.855727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:44.855820 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:44.855902 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:44.857441 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.858159 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.859726 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.860294 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.860412 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:44.863649 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:44.863655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:44.863660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:44.863694 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:44.866289 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:44.867027 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:44.867032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:44.867035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:44.867158 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:44.867577 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:44.867580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:44.867582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:44.867845 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:44.867977 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:44.868671 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 23:05:44.868891 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:05:44.870254 18757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:44.852971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.66:52699)
0620 23:05:44.853730 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:44.853774 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:44.855453 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:44.856179 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:44.856187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:44.856705 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:44.857264 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:44.857281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.858298 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.858310 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:44.859558 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:44.859570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.860484 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.860489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:44.860698 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:44.861352 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:44.861375 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:44.863448 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:44.866442 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:44.866452 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:44.866471 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:44.866891 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:44.867263 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:44.867268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:44.867272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:44.867465 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:44.867971 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:44.867979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:44.868312 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 23:05:44.868544 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 23:05:44.869097 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":298,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.67@KRBTEST.COM" created.
I20260620 23:05:44.871742 18755 heartbeater.cc:347] Connected to a master server at 127.12.2.126:41439
I20260620 23:05:44.872046 18755 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:44.872751 18755 heartbeater.cc:511] Master 127.12.2.126:41439 requested a full tablet report, sending...
I20260620 23:05:44.874202 18432 ts_manager.cc:194] Registered new tserver with Master: 174c8d98e2bc45fcb171db3de048efbf (127.12.2.66:46755)
I20260620 23:05:44.875222 18432 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:52699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab.
Entry for principal oryx/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab.
I20260620 23:05:44.893924 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:41439
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab
--principal=oryx/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:45.202883 18760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:45.203202 18760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:45.203338 18760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:45.210500 18760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:45.210604 18760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:45.210755 18760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:45.220324 18760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab
--principal=oryx/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:45.221562 18760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:45.222854 18760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:45.235267 18765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:45.235296 18768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:45.235328 18766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:45.236115 18760 server_base.cc:1034] running on GCE node
I20260620 23:05:45.236714 18760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:45.237669 18760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:45.238857 18760 hybrid_clock.cc:648] HybridClock initialized: now 1781996745238803 us; error 56 us; skew 500 ppm
I20260620 23:05:45.239181 18760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:45 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996745, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:45.242070 18760 init.cc:377] Logged in from keytab as oryx/127.12.2.67@KRBTEST.COM (short username oryx)
I20260620 23:05:45.243474 18760 webserver.cc:460] Webserver started at http://127.12.2.67:43639/ using document root <none> and password file <none>
I20260620 23:05:45.244102 18760 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:45.244210 18760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:45.244484 18760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:45.247259 18760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "f8641366ac034f14a249b162e30a5703"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-7g8g"
server_key: "bedb7de7fbfd4c28b4138af2421efde7"
server_key_iv: "7c9b61c564d25f6a65fea0d77ba42942"
server_key_version: "encryptionkey@0"
I20260620 23:05:45.247990 18760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "f8641366ac034f14a249b162e30a5703"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-7g8g"
server_key: "bedb7de7fbfd4c28b4138af2421efde7"
server_key_iv: "7c9b61c564d25f6a65fea0d77ba42942"
server_key_version: "encryptionkey@0"
I20260620 23:05:45.252910 18760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:05:45.256148 18775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:45.257429 18760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:05:45.257565 18760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "f8641366ac034f14a249b162e30a5703"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-7g8g"
server_key: "bedb7de7fbfd4c28b4138af2421efde7"
server_key_iv: "7c9b61c564d25f6a65fea0d77ba42942"
server_key_version: "encryptionkey@0"
I20260620 23:05:45.257741 18760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:45.278672 18760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:45.282696 18760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:45.282980 18760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:45.283908 18760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:45.285174 18760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:45.285252 18760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:45.285370 18760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:45.285421 18760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:45.318291 18760 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:41215
I20260620 23:05:45.318352 18887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:41215 every 8 connection(s)
I20260620 23:05:45.319597 18760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:45.320909 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 18760
I20260620 23:05:45.321085 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:45.321610 12297 external_mini_cluster.cc:1428] Setting key 94f157cdd1d766029e39a0d86834d7cd
Jun 20 23:05:45 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996745, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:45.338785 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.322238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:53487 (local address 127.12.2.126:41439)
0620 23:05:45.322417 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:45.322422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:45.323820 (+ 1398us) server_negotiation.cc:407] Connection header received
0620 23:05:45.325717 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:45.325722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:45.325806 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:45.325889 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:45.327408 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.328143 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.329621 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.330019 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.330124 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.333321 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:45.333326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:45.333329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:45.333362 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:45.335830 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.336575 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.336579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.336581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.336690 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.337143 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.337147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.337149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.337433 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:45.337555 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:45.338238 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 23:05:45.338452 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:05:45.339741 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.322888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.67:53487)
0620 23:05:45.323678 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:45.323729 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:45.325473 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:45.326139 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:45.326147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:45.326632 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:45.327247 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.327272 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.328276 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.328280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.329491 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.329502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.330228 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.330236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.330453 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.331077 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:45.331098 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:45.333118 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:45.335950 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.335956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.335969 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.336473 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.336804 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.336810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.336814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.337039 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.337548 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:45.337555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:45.337891 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 23:05:45.338122 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 23:05:45.338627 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":296,"threads_started":1}
I20260620 23:05:45.340888 18888 heartbeater.cc:347] Connected to a master server at 127.12.2.126:41439
I20260620 23:05:45.341140 18888 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:45.341814 18888 heartbeater.cc:511] Master 127.12.2.126:41439 requested a full tablet report, sending...
I20260620 23:05:45.343084 18432 ts_manager.cc:194] Registered new tserver with Master: f8641366ac034f14a249b162e30a5703 (127.12.2.67:41215)
I20260620 23:05:45.343864 18432 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:53487
I20260620 23:05:45.348766 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.2.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.125.keytab.
Entry for principal oryx/127.12.2.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.125.keytab.
I20260620 23:05:45.391342 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.125:42515
--webserver_interface=127.12.2.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.125.keytab
--principal=oryx/127.12.2.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:41439,127.12.2.125:42515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:05:45.408980 18621 heartbeater.cc:503] Master 127.12.2.126:41439 was elected leader, sending a full tablet report...
W20260620 23:05:45.694763 18893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:45.695061 18893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:45.695192 18893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:45.702389 18893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:45.702489 18893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:45.702559 18893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:45.702634 18893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:45.702692 18893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:45.712425 18893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:41439,127.12.2.125:42515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.125.keytab
--principal=oryx/127.12.2.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.125:42515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.2.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
W20260620 23:05:45.712785 18893 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.2.126:41439,127.12.2.125:42515'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 23:05:45.713864 18893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:45.715194 18893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:45.726325 18898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:45.726300 18899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:45.726277 18901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:45.727473 18893 server_base.cc:1034] running on GCE node
I20260620 23:05:45.728273 18893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:45.729596 18893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:45.730827 18893 hybrid_clock.cc:648] HybridClock initialized: now 1781996745730749 us; error 40 us; skew 500 ppm
I20260620 23:05:45.731187 18893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:45 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996745, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:45.737807 18893 init.cc:377] Logged in from keytab as oryx/127.12.2.125@KRBTEST.COM (short username oryx)
I20260620 23:05:45.739238 18893 webserver.cc:460] Webserver started at http://127.12.2.125:39723/ using document root <none> and password file <none>
I20260620 23:05:45.740017 18893 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:45.740118 18893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:45.740411 18893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:45.743341 18893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data/instance:
uuid: "8904843a27ca47bb81ab9e2ddf21123b"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-7g8g"
server_key: "b5313c2e9e83fd070c1652280d5651a2"
server_key_iv: "edc888d2a0afa51fe6d73a576a085e2c"
server_key_version: "encryptionkey@0"
I20260620 23:05:45.744153 18893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/wal/instance:
uuid: "8904843a27ca47bb81ab9e2ddf21123b"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-7g8g"
server_key: "b5313c2e9e83fd070c1652280d5651a2"
server_key_iv: "edc888d2a0afa51fe6d73a576a085e2c"
server_key_version: "encryptionkey@0"
I20260620 23:05:45.749130 18893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:05:45.752409 18908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:45.753753 18893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:05:45.753898 18893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/wal
uuid: "8904843a27ca47bb81ab9e2ddf21123b"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-7g8g"
server_key: "b5313c2e9e83fd070c1652280d5651a2"
server_key_iv: "edc888d2a0afa51fe6d73a576a085e2c"
server_key_version: "encryptionkey@0"
I20260620 23:05:45.754078 18893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:45.771184 18893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:45.774914 18893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:45.775177 18893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:45.791186 18893 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.125:42515
I20260620 23:05:45.791206 18959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.125:42515 every 8 connection(s)
I20260620 23:05:45.792460 18893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
I20260620 23:05:45.795868 18960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:45.799489 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 18893
I20260620 23:05:45.802652 18960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:05:45 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.2.125@KRBTEST.COM
Jun 20 23:05:45 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996745, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.125@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:45.820711 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.806066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.12.2.126:41439)
0620 23:05:45.806214 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:45.806218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:45.806956 (+ 738us) server_negotiation.cc:407] Connection header received
0620 23:05:45.807114 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:45.807120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:45.807190 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:45.807266 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:45.808451 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.809178 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.810790 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.811239 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.811362 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.815261 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:45.815269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:45.815274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:45.815311 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:45.817801 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.818438 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.818442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.818445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.818572 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.819020 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.819025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.819028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.819400 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:45.819526 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:45.820165 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 23:05:45.820373 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:05:45.822082 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.806151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:44512)
0620 23:05:45.806813 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:45.806845 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:45.807001 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:45.807350 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:45.807357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:45.808015 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:45.808305 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.808323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.809321 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.809325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.810648 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.810662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.811453 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.811461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.811636 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.812267 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:45.812297 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:45.815071 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:45.817919 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.817931 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.817943 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.818331 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.818671 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.818676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.818680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.818882 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.819525 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:45.819533 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:45.819804 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 23:05:45.821110 (+ 1306us) client_negotiation.cc:240] Negotiation successful
0620 23:05:45.821375 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":313,"threads_started":1}
I20260620 23:05:45.822099 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.801509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.12.2.125:42515)
0620 23:05:45.802221 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:45.802232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:45.802271 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:05:45.803812 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:45.803833 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:45.804231 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:45.804570 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:45.805967 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.807679 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.809103 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.809825 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.809958 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.814572 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:45.814579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:45.814590 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:45.814617 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:45.817839 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.818629 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.818637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.818641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.818816 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.819257 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.819262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.819265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.819872 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:45.820152 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:45.820479 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 23:05:45.820873 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"spinlock_wait_cycles":18304,"thread_start_us":325,"threads_started":1}
I20260620 23:05:45.824946 18960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:05:45 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996745, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.125@KRBTEST.COM for oryx/127.12.2.125@KRBTEST.COM
I20260620 23:05:45.836258 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.825521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42515 (local address 127.0.0.1:49154)
0620 23:05:45.825741 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:45.825756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:45.825837 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:45.826091 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:45.826094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:45.826588 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:45.826711 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.826718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.827725 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.827729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.828626 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.828638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.829256 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.829261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.829390 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.829880 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:45.829896 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:45.831871 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:45.834342 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.834345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.834348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.834713 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.835018 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.835021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.835023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.835141 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.835565 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:45.835568 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:45.835706 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 23:05:45.835820 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:05:45.836017 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 23:05:45.836377 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.825662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.12.2.125:42515)
0620 23:05:45.825826 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:45.825831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:45.825855 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:05:45.825936 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:45.825939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:45.825986 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:45.826056 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:45.826826 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.827553 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.828749 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.829115 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.829206 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.832019 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:45.832023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:45.832026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:45.832051 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:45.834226 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.834801 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.834804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.834807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.834916 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.835228 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.835231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.835233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.835479 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:45.835578 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:45.835902 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 23:05:45.836133 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":41}
I20260620 23:05:45.840801 18960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: Bootstrap starting.
I20260620 23:05:45.843770 18960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:45.844731 18960 log.cc:824] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:45.847213 18960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: No bootstrap required, opened a new log
I20260620 23:05:45.854101 18960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } }
I20260620 23:05:45.854403 18960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:45.854471 18960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8904843a27ca47bb81ab9e2ddf21123b, State: Initialized, Role: FOLLOWER
I20260620 23:05:45.854981 18960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } }
I20260620 23:05:45.855731 18966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } } }
I20260620 23:05:45.856005 18966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:05:45.856647 18960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:05:45.865197 18977 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:05:45.865325 18977 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:05:45.867513 18893 master_runner.cc:428] Detected that this master 8904843a27ca47bb81ab9e2ddf21123b is joining an existing cluster
I20260620 23:05:45.867596 18893 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.2.125:42515
I20260620 23:05:45.878113 18755 heartbeater.cc:503] Master 127.12.2.126:41439 was elected leader, sending a full tablet report...
I20260620 23:05:45.887188 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.870972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.12.2.126:41439)
0620 23:05:45.871131 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:45.871137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:45.872640 (+ 1503us) server_negotiation.cc:407] Connection header received
0620 23:05:45.872819 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:45.872824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:45.872898 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:45.872994 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:45.874206 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.875131 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.876966 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.877576 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.877662 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.881245 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:45.881251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:45.881256 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:45.881289 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:45.884251 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.884990 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.884996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.884999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.885160 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.885542 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.885547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.885550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.885873 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:45.885988 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:45.886588 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 23:05:45.886836 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:05:45.887529 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.871241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:44520)
0620 23:05:45.872524 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:45.872541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:45.872695 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:45.873028 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:45.873034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:45.873746 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:45.874056 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.874067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.875265 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.875270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.876807 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.876821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.877928 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.877934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.878107 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.879911 (+ 1804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:45.879938 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:45.881052 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:45.884408 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.884413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.884417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.884846 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.885292 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.885295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.885297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.885427 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.886080 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:45.886085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:45.886242 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 23:05:45.886437 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 23:05:45.886741 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"mutex_wait_us":28,"spinlock_wait_cycles":8448,"thread_start_us":988,"threads_started":1}
I20260620 23:05:45.888159 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.871147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42515 (local address 127.0.0.1:49168)
0620 23:05:45.871847 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:45.871868 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:45.872046 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:45.872488 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:45.872491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:45.873122 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:45.873448 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.873457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.874581 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.874586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.876747 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.876760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.879082 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.879089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.879277 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.879852 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:45.880080 (+ 228us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:45.881281 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:45.885518 (+ 4237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.885522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.885526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.885994 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.886393 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.886398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.886402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.886546 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.887221 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:45.887228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:45.887365 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 23:05:45.887487 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:05:45.887759 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":4,"thread_start_us":659,"threads_started":1}
I20260620 23:05:45.888365 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.871599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.12.2.125:42515)
0620 23:05:45.871808 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:45.871814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:45.871947 (+ 133us) server_negotiation.cc:407] Connection header received
0620 23:05:45.872177 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:45.872182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:45.872246 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:45.872308 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:45.873570 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.874405 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.876886 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.877611 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.877717 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.881441 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:45.881447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:45.881451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:45.881487 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:45.885400 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.886120 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.886125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.886129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.886280 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.886653 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.886658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.886661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.887035 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:45.887155 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:45.887614 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 23:05:45.887907 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":57}
WARNING: no policy specified for oryx/127.12.2.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.124@KRBTEST.COM" created.
Jun 20 23:05:45 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.126@KRBTEST.COM for oryx/127.12.2.125@KRBTEST.COM
I20260620 23:05:45.908672 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.893288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42515 (local address 127.0.0.1:49172)
0620 23:05:45.893844 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:45.893875 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:45.894036 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:45.894602 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:45.894607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:45.895255 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:45.895453 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.895462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.896427 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.896433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.897548 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.897560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.898298 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.898304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.898479 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.899109 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:45.899129 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:45.902298 (+ 3169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:45.905918 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.905932 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.905940 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.906440 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.906832 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.906847 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.906851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.907029 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.907641 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:45.907650 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:45.907831 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 23:05:45.908031 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 23:05:45.908345 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260620 23:05:45.909363 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.893204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49172 (local address 127.12.2.125:42515)
0620 23:05:45.893338 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:45.893343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:45.893955 (+ 612us) server_negotiation.cc:407] Connection header received
0620 23:05:45.894240 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:45.894246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:45.894347 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:45.894402 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:45.895605 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.896312 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.897674 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.898154 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.898256 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.902533 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:45.902539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:45.902544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:45.902580 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:45.905753 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.906574 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.906580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.906583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.906722 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.907148 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.907152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.907156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.907453 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:45.907575 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:45.908970 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0620 23:05:45.909146 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:05:45.911087 18432 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.2.125:42515: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b"
member_type: NON_VOTER
last_known_addr {
host: "127.12.2.125"
port: 42515
}
attrs {
promote: true
}
}
dest_uuid: "07a99eb094dc4ae295ad268a70895388"
cas_config_opid_index: -1
I20260620 23:05:45.912019 18432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } }
I20260620 23:05:45.925127 18989 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8904843a27ca47bb81ab9e2ddf21123b (127.12.2.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } } }
I20260620 23:05:45.926302 18935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:45.929834 18935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 FOLLOWER]: Refusing update from remote peer 07a99eb094dc4ae295ad268a70895388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 23:05:45.929947 18989 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.2.125:42515
I20260620 23:05:45.930754 18893 master.cc:537] Master@127.12.2.125:42515 shutting down...
I20260620 23:05:45.930517 18990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } } }
I20260620 23:05:45.930873 18990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:45.932729 18989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.124.keytab.
Entry for principal oryx/127.12.2.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.124.keytab.
I20260620 23:05:45.935753 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.124:33779
--webserver_interface=127.12.2.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.124.keytab
--principal=oryx/127.12.2.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:41439,127.12.2.125:42515,127.12.2.124:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:05:45.941781 18893 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:45.942162 18893 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:45.942343 18893 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: stopping tablet replica
W20260620 23:05:45.947639 18417 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from unknown peer (error 108)
I20260620 23:05:45.949370 18893 master.cc:552] Master@127.12.2.125:42515 shutdown complete.
I20260620 23:05:45.949458 18893 master_runner.cc:305] Clearing existing system tablet
I20260620 23:05:45.949887 18893 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:45.949977 18893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:45.959579 18893 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.005s
I20260620 23:05:45.962246 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.961584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42515 (local address 127.0.0.1:49180)
0620 23:05:45.961934 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:45.962063 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
W20260620 23:05:45.962631 18417 consensus_peers.cc:487] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 -> Peer 8904843a27ca47bb81ab9e2ddf21123b (127.12.2.125:42515): Couldn't send request to peer 8904843a27ca47bb81ab9e2ddf21123b. Status: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:05:45.963167 18996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:45.963469 18893 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:45.963721 18893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:45.963805 18893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/wal
uuid: "8904843a27ca47bb81ab9e2ddf21123b"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-7g8g"
server_key: "b5313c2e9e83fd070c1652280d5651a2"
server_key_iv: "edc888d2a0afa51fe6d73a576a085e2c"
server_key_version: "encryptionkey@0"
I20260620 23:05:45.967309 18893 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:45.972009 18893 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:05:45.972154 18893 log.cc:1196] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 23:05:45.972518 18893 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: Deleting consensus metadata
I20260620 23:05:45.972736 18893 master_runner.cc:315] Copying system tablet from 127.12.2.126:41439
I20260620 23:05:45.976415 18893 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: tablet copy: Beginning tablet copy session from remote peer at address 127.12.2.126:41439
I20260620 23:05:45.988168 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.977125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.12.2.126:41439)
0620 23:05:45.977307 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:45.977314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:45.977782 (+ 468us) server_negotiation.cc:407] Connection header received
0620 23:05:45.978022 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:45.978028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:45.978109 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:45.978190 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:45.979004 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.979709 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.980913 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.981309 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.981417 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.983095 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:45.983100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:45.983103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:45.983135 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:45.985558 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.986176 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.986181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.986185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.986301 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:45.986653 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:45.986659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:45.986663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:45.987002 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:45.987133 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:45.987527 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 23:05:45.987744 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:05:45.988161 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.977104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:44536)
0620 23:05:45.977668 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:45.977689 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:45.977892 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:45.978230 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:45.978234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:45.978617 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:45.978817 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.978827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.979862 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.979868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.980776 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:45.980788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:45.981454 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:45.981460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:45.981621 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:45.982183 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:45.982201 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:45.982880 (+ 679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:45.985695 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.985701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.985705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.986065 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.986405 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:45.986410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:45.986414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:45.986555 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:45.987108 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:45.987113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:45.987276 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 23:05:45.987401 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 23:05:45.987709 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":359,"threads_started":1}
I20260620 23:05:45.989230 18452 tablet_copy_service.cc:140] P 07a99eb094dc4ae295ad268a70895388: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8904843a27ca47bb81ab9e2ddf21123b ({username='oryx', principal='oryx/127.12.2.125@KRBTEST.COM'} at 127.0.0.1:44536)
I20260620 23:05:45.989470 18452 tablet_copy_service.cc:161] P 07a99eb094dc4ae295ad268a70895388: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8904843a27ca47bb81ab9e2ddf21123b at {username='oryx', principal='oryx/127.12.2.125@KRBTEST.COM'} at 127.0.0.1:44536: session id = 8904843a27ca47bb81ab9e2ddf21123b-00000000000000000000000000000000
I20260620 23:05:45.992854 18452 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:05:45.994727 18893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:05:45.999436 18893 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: tablet copy: Starting download of 0 data blocks...
I20260620 23:05:45.999712 18893 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: tablet copy: Starting download of 1 WAL segments...
I20260620 23:05:46.002081 18893 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:05:46.005383 18452 tablet_copy_service.cc:342] P 07a99eb094dc4ae295ad268a70895388: Request end of tablet copy session 8904843a27ca47bb81ab9e2ddf21123b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.2.125@KRBTEST.COM'} at 127.0.0.1:44536
I20260620 23:05:46.005571 18452 tablet_copy_service.cc:433] P 07a99eb094dc4ae295ad268a70895388: ending tablet copy session 8904843a27ca47bb81ab9e2ddf21123b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8904843a27ca47bb81ab9e2ddf21123b
I20260620 23:05:46.015232 18893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:46.017632 19003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:46.023814 19004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:46.031451 18893 server_base.cc:1034] running on GCE node
W20260620 23:05:46.031769 19006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:46.032266 18893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:46.032871 18893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:46.034078 18893 hybrid_clock.cc:648] HybridClock initialized: now 1781996746034025 us; error 33 us; skew 500 ppm
I20260620 23:05:46.034291 18893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:46 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996746, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:46.036150 18893 init.cc:377] Logged in from keytab as oryx/127.12.2.125@KRBTEST.COM (short username oryx)
I20260620 23:05:46.037297 18893 webserver.cc:460] Webserver started at http://127.12.2.125:33765/ using document root <none> and password file <none>
I20260620 23:05:46.037613 18893 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:46.037684 18893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:46.040103 18893 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:05:46.041600 19013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:46.041961 18893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:46.042083 18893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/wal
uuid: "8904843a27ca47bb81ab9e2ddf21123b"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-7g8g"
server_key: "b5313c2e9e83fd070c1652280d5651a2"
server_key_iv: "edc888d2a0afa51fe6d73a576a085e2c"
server_key_version: "encryptionkey@0"
I20260620 23:05:46.042217 18893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:46.054591 18893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:46.057789 18893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:46.224018 19064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.125:42515 every 8 connection(s)
I20260620 23:05:46.227077 18893 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.125:42515
I20260620 23:05:46.227880 18893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
I20260620 23:05:46.236004 19065 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:05:46.243795 19065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: Bootstrap starting.
I20260620 23:05:46.271180 19065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:05:46.271631 19065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: Bootstrap complete.
I20260620 23:05:46.272518 19065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } }
I20260620 23:05:46.272728 19065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8904843a27ca47bb81ab9e2ddf21123b, State: Initialized, Role: LEARNER
I20260620 23:05:46.272933 19065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } }
I20260620 23:05:46.275556 19065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:46.275537 19068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } } }
I20260620 23:05:46.276069 19068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: This master's current role is: LEARNER
I20260620 23:05:46.286967 19079 catalog_manager.cc:1259] Loaded cluster ID: 1befd225d77342798bc52be4cecf56e2
I20260620 23:05:46.287071 19079 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: loading cluster ID for follower catalog manager: success
I20260620 23:05:46.290489 19079 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: acquiring CA information for follower catalog manager: success
I20260620 23:05:46.293015 19079 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:05:46.346217 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.337579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42515 (local address 127.0.0.1:49196)
0620 23:05:46.337845 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:46.337865 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:46.337994 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:46.338964 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:46.338970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:46.339006 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:46.339199 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.339212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.341451 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.341457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.343598 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.343616 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.345371 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.345378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.345581 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.345650 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 23:05:46.345754 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:05:46.345918 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260620 23:05:46.346366 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.337730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.12.2.125:42515)
0620 23:05:46.338521 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.338527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.338546 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:05:46.338622 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.338627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.338718 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.338850 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:46.339399 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.341282 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.343790 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.345173 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.345312 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.345589 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.345820 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.345984 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":460,"threads_started":1}
I20260620 23:05:46.346505 18888 heartbeater.cc:503] Master 127.12.2.126:41439 was elected leader, sending a full tablet report...
I20260620 23:05:46.347689 19040 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 23:05:46.348341 19068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a99eb094dc4ae295ad268a70895388. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } } }
I20260620 23:05:46.348589 19068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: This master's current role is: LEARNER
W20260620 23:05:46.373669 18993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:46.373971 18993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:46.374104 18993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:46.383205 18993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:46.383319 18993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:46.383394 18993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:46.383462 18993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:46.383517 18993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:46.393338 18993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:41439,127.12.2.125:42515,127.12.2.124:33779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.124.keytab
--principal=oryx/127.12.2.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.124:33779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.2.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:46.394732 18993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:46.396054 18993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:46.407560 19085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:46.407536 19086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:46.407640 19088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:46.408703 18993 server_base.cc:1034] running on GCE node
I20260620 23:05:46.409418 18993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:46.410753 18993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:46.411952 18993 hybrid_clock.cc:648] HybridClock initialized: now 1781996746411894 us; error 49 us; skew 500 ppm
I20260620 23:05:46.412326 18993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:46 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996746, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:46.415313 18993 init.cc:377] Logged in from keytab as oryx/127.12.2.124@KRBTEST.COM (short username oryx)
I20260620 23:05:46.416678 18993 webserver.cc:460] Webserver started at http://127.12.2.124:34961/ using document root <none> and password file <none>
I20260620 23:05:46.417289 18993 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:46.417397 18993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:46.417632 18993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:46.420414 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data/instance:
uuid: "3f9a7aaaa9174ba6a11acf9635519a8d"
format_stamp: "Formatted at 2026-06-20 23:05:46 on dist-test-slave-7g8g"
server_key: "6924fd51fbc9a4273267d701fa14d3d0"
server_key_iv: "ce99f7a7a5cdd3786d948beabc23a1fc"
server_key_version: "encryptionkey@0"
I20260620 23:05:46.421092 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/wal/instance:
uuid: "3f9a7aaaa9174ba6a11acf9635519a8d"
format_stamp: "Formatted at 2026-06-20 23:05:46 on dist-test-slave-7g8g"
server_key: "6924fd51fbc9a4273267d701fa14d3d0"
server_key_iv: "ce99f7a7a5cdd3786d948beabc23a1fc"
server_key_version: "encryptionkey@0"
I20260620 23:05:46.425920 18993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:05:46.429276 19095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:46.430565 18993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:05:46.430709 18993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/wal
uuid: "3f9a7aaaa9174ba6a11acf9635519a8d"
format_stamp: "Formatted at 2026-06-20 23:05:46 on dist-test-slave-7g8g"
server_key: "6924fd51fbc9a4273267d701fa14d3d0"
server_key_iv: "ce99f7a7a5cdd3786d948beabc23a1fc"
server_key_version: "encryptionkey@0"
I20260620 23:05:46.430882 18993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:46.448446 18993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:46.452896 18993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:46.453161 18993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:46.471230 18993 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.124:33779
I20260620 23:05:46.471232 19146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.124:33779 every 8 connection(s)
I20260620 23:05:46.472653 18993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
I20260620 23:05:46.476101 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 18993
I20260620 23:05:46.476398 19147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:46.485489 19147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:05:46 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.2.124@KRBTEST.COM
Jun 20 23:05:46 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996746, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.124@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:46.498875 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.478305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.12.2.124:33779)
0620 23:05:46.479137 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.479152 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.479204 (+ 52us) server_negotiation.cc:407] Connection header received
0620 23:05:46.480971 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.480999 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.481567 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.482094 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.483584 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.484833 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.486047 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.486733 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.486898 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.491323 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.491330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.491340 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.491373 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.494877 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.495718 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.495726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.495735 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.495906 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.496242 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.496247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.496250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.496729 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.497005 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.497429 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.497822 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":331,"threads_started":1}
I20260620 23:05:46.501295 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.487876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.12.2.126:41439)
0620 23:05:46.488062 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.488067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.488489 (+ 422us) server_negotiation.cc:407] Connection header received
0620 23:05:46.488642 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.488648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.488715 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.488794 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.489815 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.490677 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.492098 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.492694 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.492858 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.496329 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.496334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.496338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.496370 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.498677 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.499473 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.499479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.499483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.499603 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.499919 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.499924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.499928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.500224 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.500347 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.500798 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.500998 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:05:46.501360 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.487764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:44540)
0620 23:05:46.488366 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:46.488399 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:46.488533 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:46.488862 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:46.488868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:46.489450 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:46.489672 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.489680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.490853 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.490857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.491981 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.491996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.492913 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.492919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.493056 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.493580 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:46.493596 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:46.496157 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:46.498799 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.498808 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.498815 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.499370 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.499699 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.499704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.499708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.499818 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.500347 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:46.500354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:46.500502 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 23:05:46.500660 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 23:05:46.500945 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
I20260620 23:05:46.502699 19147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:05:46 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996746, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.124@KRBTEST.COM for oryx/127.12.2.125@KRBTEST.COM
I20260620 23:05:46.513793 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.503387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.12.2.125:42515)
0620 23:05:46.503541 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.503547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.503606 (+ 59us) server_negotiation.cc:407] Connection header received
0620 23:05:46.503759 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.503762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.503817 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.503901 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.504479 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.505157 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.506277 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.506674 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.506772 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.509293 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.509298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.509301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.509329 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.511650 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.512197 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.512202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.512205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.512319 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.512621 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.512625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.512628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.512878 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.512997 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.513299 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.513495 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:05:46.513787 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.503374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42515 (local address 127.0.0.1:49210)
0620 23:05:46.503543 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:46.503561 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:46.503657 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:46.503927 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:46.503932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:46.504229 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:46.504357 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.504365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.505268 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.505273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.506155 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.506165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.506797 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.506802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.507004 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.507422 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:46.507436 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:46.509138 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:46.511767 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.511772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.511776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.512097 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.512412 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.512417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.512420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.512531 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.512979 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:46.512983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:46.513101 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 23:05:46.513207 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:05:46.513473 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 23:05:46.514765 19147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:05:46 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996746, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.124@KRBTEST.COM for oryx/127.12.2.124@KRBTEST.COM
I20260620 23:05:46.525792 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.515886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:33779 (local address 127.0.0.1:49832)
0620 23:05:46.516030 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:46.516043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:46.516134 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:46.516377 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:46.516381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:46.516686 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:46.516802 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.516809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.517656 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.517661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.518440 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.518450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.519077 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.519083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.519264 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.519682 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:46.519696 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:46.521410 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:46.523759 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.523764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.523768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.524067 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.524353 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.524358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.524362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.524486 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.524921 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:46.524925 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:46.525053 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 23:05:46.525158 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:05:46.525395 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 23:05:46.525831 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.515829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.12.2.124:33779)
0620 23:05:46.515980 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.515985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.516098 (+ 113us) server_negotiation.cc:407] Connection header received
0620 23:05:46.516220 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.516225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.516271 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.516339 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.516916 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.517550 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.518552 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.518923 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.519050 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.521553 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.521557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.521560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.521585 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.523650 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.524160 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.524165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.524168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.524257 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.524576 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.524580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.524584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.524816 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.524940 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.525241 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.525538 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":47}
I20260620 23:05:46.530248 19147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: Bootstrap starting.
I20260620 23:05:46.533221 19147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:46.534610 19147 log.cc:824] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:46.537228 19147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: No bootstrap required, opened a new log
I20260620 23:05:46.544083 19147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } }
I20260620 23:05:46.544382 19147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:46.544461 19147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9a7aaaa9174ba6a11acf9635519a8d, State: Initialized, Role: FOLLOWER
I20260620 23:05:46.544935 19147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } }
I20260620 23:05:46.545655 19153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } } }
I20260620 23:05:46.545981 19153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:05:46.546572 19147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:46.553258 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 18493
W20260620 23:05:46.555616 19164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:05:46.555743 19164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:05:46.560418 18993 master_runner.cc:428] Detected that this master 3f9a7aaaa9174ba6a11acf9635519a8d is joining an existing cluster
I20260620 23:05:46.560559 18993 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.2.124:33779
Jun 20 23:05:46 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.2.65@KRBTEST.COM
I20260620 23:05:46.575384 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.554429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.12.2.65:39893)
0620 23:05:46.555061 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.555069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.555096 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:05:46.555161 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.555169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.555432 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.556661 (+ 1229us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.557271 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.558819 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.560143 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.561344 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.563045 (+ 1701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.566749 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.566763 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.566780 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.566821 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.570450 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.571309 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.571318 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.571321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.571463 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.572286 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.572291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.572294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.572701 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.574035 (+ 1334us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.574193 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.574455 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":308,"threads_started":1}
I20260620 23:05:46.576745 18616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:46.599648 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.565236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.12.2.125:42515)
0620 23:05:46.565408 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.565413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.576272 (+ 10859us) server_negotiation.cc:407] Connection header received
0620 23:05:46.579977 (+ 3705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.579983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.580063 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.580116 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.581284 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.582366 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.584247 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.584853 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.584938 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.589467 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.589474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.589479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.589518 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.592711 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.597184 (+ 4473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.597189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.597193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.597353 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.597833 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.597838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.597841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.598184 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.598320 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.598973 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.599304 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:05:46.600183 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.564702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.12.2.126:41439)
0620 23:05:46.564903 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.564909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.581634 (+ 16725us) server_negotiation.cc:407] Connection header received
0620 23:05:46.581765 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.581770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.581846 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.581935 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.583546 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.584608 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.586868 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.587517 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.587595 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.592270 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.592277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.592282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.592323 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.595582 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.596691 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.596697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.596701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.596871 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.597934 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.597940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.597944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.598288 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.598419 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.599576 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.599805 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:05:46.601230 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.565225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:44546)
0620 23:05:46.581538 (+ 16313us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:46.581552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:46.581663 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:46.582097 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:46.582101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:46.583040 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:46.583373 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.583385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.584803 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.584808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.586696 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.586710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.587850 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.587856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.588286 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.588944 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:46.588967 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:46.590641 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:46.595731 (+ 5090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.595736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.595740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.596566 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.597014 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.597019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.597023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.597195 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.598494 (+ 1299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:46.598499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:46.598748 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 23:05:46.600415 (+ 1667us) client_negotiation.cc:240] Negotiation successful
0620 23:05:46.600757 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16206,"mutex_wait_us":265,"thread_start_us":15918,"threads_started":1}
I20260620 23:05:46.601673 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.564766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42515 (local address 127.0.0.1:49214)
0620 23:05:46.576130 (+ 11364us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:46.576197 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:46.579686 (+ 3489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:46.580335 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:46.580340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:46.580849 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:46.581146 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.581159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.582474 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.582479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.584097 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.584110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.585206 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.585212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.585391 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.588139 (+ 2748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:46.588162 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:46.589264 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:46.596090 (+ 6826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.596097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.596101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.597051 (+ 950us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.597515 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.597520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.597524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.597705 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.598393 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:46.598398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:46.598595 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 23:05:46.600166 (+ 1571us) client_negotiation.cc:240] Negotiation successful
0620 23:05:46.600943 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11171,"mutex_wait_us":92,"thread_start_us":16046,"threads_started":1}
I20260620 23:05:46.605321 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.568377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.12.2.124:33779)
0620 23:05:46.568552 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.568557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.582060 (+ 13503us) server_negotiation.cc:407] Connection header received
0620 23:05:46.582243 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.582247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.582305 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.582372 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.583693 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.584771 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.586699 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.587910 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.588026 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.593028 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.593034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.593039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.593071 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.596528 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.598966 (+ 2438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.598971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.598974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.599117 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.603790 (+ 4673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.603795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.603799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.604148 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.604278 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.604728 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.604940 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":26880}
I20260620 23:05:46.606796 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.568474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:33779 (local address 127.0.0.1:49844)
0620 23:05:46.581989 (+ 13515us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:46.582004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:46.582124 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:46.582545 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:46.582548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:46.583246 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:46.583562 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.583573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.584989 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.584994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.586566 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.586580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.589219 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.589223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.589373 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.591017 (+ 1644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:46.591052 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:46.592055 (+ 1003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:46.597295 (+ 5240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.597300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.597304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.597787 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.599236 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.599240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.599243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.599369 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.604346 (+ 4977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:46.604351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:46.604483 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 23:05:46.605648 (+ 1165us) client_negotiation.cc:240] Negotiation successful
0620 23:05:46.605923 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13397,"thread_start_us":6716,"threads_started":1}
Jun 20 23:05:46 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.126@KRBTEST.COM for oryx/127.12.2.124@KRBTEST.COM
I20260620 23:05:46.623077 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.606473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:33779 (local address 127.0.0.1:49848)
0620 23:05:46.606808 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:46.606827 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:46.606925 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:46.607782 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:46.607788 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:46.608194 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:46.608375 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.608386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.610205 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.610212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.611596 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.611611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.612598 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.612605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.612774 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.613395 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:46.613418 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:46.616136 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:46.619984 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.619990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.619994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.620459 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.620911 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.620916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.620919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.621076 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.622099 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:46.622104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:46.622265 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 23:05:46.622400 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 23:05:46.622671 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260620 23:05:46.623119 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.607136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.12.2.124:33779)
0620 23:05:46.607300 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.607306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.607322 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:05:46.607406 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.607411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.607509 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.607596 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.608535 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.610059 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.611741 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.612448 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.612561 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.616334 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.616341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.616346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.616385 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.619826 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.620585 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.620591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.620617 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.620778 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.621214 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.621219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.621223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.621970 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.622110 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.622518 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.622752 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:05:46.624759 18432 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.2.124:33779: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d"
member_type: NON_VOTER
last_known_addr {
host: "127.12.2.124"
port: 33779
}
attrs {
promote: true
}
}
dest_uuid: "07a99eb094dc4ae295ad268a70895388"
cas_config_opid_index: 5
I20260620 23:05:46.625763 18432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } }
I20260620 23:05:46.627540 18990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3f9a7aaaa9174ba6a11acf9635519a8d (127.12.2.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:46.628147 19040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 LEARNER]: Refusing update from remote peer 07a99eb094dc4ae295ad268a70895388: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:05:46.629518 19122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:46.633244 19122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 FOLLOWER]: Refusing update from remote peer 07a99eb094dc4ae295ad268a70895388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:05:46.643898 19177 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 23:05:46.648622 18990 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.2.124:33779
I20260620 23:05:46.649246 18993 master.cc:537] Master@127.12.2.124:33779 shutting down...
I20260620 23:05:46.663126 19177 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:46.670146 19176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:46.670550 19176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:46.671269 19040 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3f9a7aaaa9174ba6a11acf9635519a8d (127.12.2.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:46.675557 19068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: NON_VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: true } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:46.675849 19068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: This master's current role is: LEARNER
I20260620 23:05:46.676767 19176 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388: attempting to promote NON_VOTER 8904843a27ca47bb81ab9e2ddf21123b to VOTER
I20260620 23:05:46.677596 19176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } }
I20260620 23:05:46.679903 19040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 LEARNER]: Refusing update from remote peer 07a99eb094dc4ae295ad268a70895388: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 23:05:46.680442 19176 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 23:05:46.684974 19040 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8904843a27ca47bb81ab9e2ddf21123b (127.12.2.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:46.685936 18993 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:46.682832 19175 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8904843a27ca47bb81ab9e2ddf21123b (127.12.2.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:46.686354 18993 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:46.686560 18993 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: stopping tablet replica
I20260620 23:05:46.689718 19068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:46.689946 19068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:05:46.690338 18416 connection.cc:535] client connection to 127.12.2.124:33779 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 23:05:46.690513 18416 consensus_peers.cc:487] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 -> Peer 3f9a7aaaa9174ba6a11acf9635519a8d (127.12.2.124:33779): Couldn't send request to peer 3f9a7aaaa9174ba6a11acf9635519a8d. Status: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:05:46.690866 19176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:46.691148 19176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:46.691401 19176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:46.691618 19176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:46.703215 18993 master.cc:552] Master@127.12.2.124:33779 shutdown complete.
I20260620 23:05:46.703299 18993 master_runner.cc:305] Clearing existing system tablet
I20260620 23:05:46.703613 18993 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:46.703732 18993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:46.705811 18993 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:05:46.707144 19182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:46.707442 18993 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:46.707713 18993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:46.707809 18993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/wal
uuid: "3f9a7aaaa9174ba6a11acf9635519a8d"
format_stamp: "Formatted at 2026-06-20 23:05:46 on dist-test-slave-7g8g"
server_key: "6924fd51fbc9a4273267d701fa14d3d0"
server_key_iv: "ce99f7a7a5cdd3786d948beabc23a1fc"
server_key_version: "encryptionkey@0"
I20260620 23:05:46.711184 18993 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:46.716132 18993 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:05:46.716274 18993 log.cc:1196] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 23:05:46.716619 18993 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: Deleting consensus metadata
I20260620 23:05:46.716827 18993 master_runner.cc:315] Copying system tablet from 127.12.2.126:41439
I20260620 23:05:46.719110 18993 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: tablet copy: Beginning tablet copy session from remote peer at address 127.12.2.126:41439
I20260620 23:05:46.723132 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 18493
I20260620 23:05:46.736403 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.719849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.12.2.126:41439)
0620 23:05:46.720101 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.720108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.720572 (+ 464us) server_negotiation.cc:407] Connection header received
0620 23:05:46.720826 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.720832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.720910 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.720993 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.722113 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.723320 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.726968 (+ 3648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.727585 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.727699 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.730111 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.730118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.730122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.730164 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.733456 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.734250 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.734256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.734260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.734409 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.734808 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.734813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.734817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.735180 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.735321 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.735772 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.736026 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 23:05:46.736413 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.719797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:44554)
0620 23:05:46.720455 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:46.720478 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:46.720682 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:46.721157 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:46.721162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:46.721682 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:46.721933 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.721951 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.725158 (+ 3207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.725165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.726775 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:46.726792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.727873 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.727879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:46.728049 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.728763 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:46.728789 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:46.729884 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:46.733608 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.733613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.733618 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.734123 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.734536 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:46.734541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:46.734545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:46.734705 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:46.735307 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:46.735312 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:46.735517 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 23:05:46.735659 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 23:05:46.735976 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":429,"threads_started":1}
I20260620 23:05:46.737457 18452 tablet_copy_service.cc:140] P 07a99eb094dc4ae295ad268a70895388: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3f9a7aaaa9174ba6a11acf9635519a8d ({username='oryx', principal='oryx/127.12.2.124@KRBTEST.COM'} at 127.0.0.1:44554)
I20260620 23:05:46.737617 18452 tablet_copy_service.cc:161] P 07a99eb094dc4ae295ad268a70895388: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3f9a7aaaa9174ba6a11acf9635519a8d at {username='oryx', principal='oryx/127.12.2.124@KRBTEST.COM'} at 127.0.0.1:44554: session id = 3f9a7aaaa9174ba6a11acf9635519a8d-00000000000000000000000000000000
I20260620 23:05:46.738582 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:39893
--local_ip_for_outbound_sockets=127.12.2.65
--tserver_master_addrs=127.12.2.126:41439
--webserver_port=40137
--webserver_interface=127.12.2.65
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab
--principal=oryx/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.2.126:41439,127.12.2.125:42515
--tserver_master_addrs=127.12.2.126:41439,127.12.2.125:42515,127.12.2.124:33779 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:05:46.741173 18452 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:05:46.743026 18993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:05:46.747603 18993 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: tablet copy: Starting download of 0 data blocks...
I20260620 23:05:46.747853 18993 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: tablet copy: Starting download of 1 WAL segments...
I20260620 23:05:46.750072 18993 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:05:46.753181 18452 tablet_copy_service.cc:342] P 07a99eb094dc4ae295ad268a70895388: Request end of tablet copy session 3f9a7aaaa9174ba6a11acf9635519a8d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.2.124@KRBTEST.COM'} at 127.0.0.1:44554
I20260620 23:05:46.753295 18452 tablet_copy_service.cc:433] P 07a99eb094dc4ae295ad268a70895388: ending tablet copy session 3f9a7aaaa9174ba6a11acf9635519a8d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3f9a7aaaa9174ba6a11acf9635519a8d
I20260620 23:05:46.762823 18993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:46.765232 19190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:46.768392 19191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:46.775377 19193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:46.776078 18993 server_base.cc:1034] running on GCE node
I20260620 23:05:46.776369 18993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:46.776988 18993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:46.778158 18993 hybrid_clock.cc:648] HybridClock initialized: now 1781996746778119 us; error 33 us; skew 500 ppm
I20260620 23:05:46.778369 18993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:46 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996746, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:46.780123 18993 init.cc:377] Logged in from keytab as oryx/127.12.2.124@KRBTEST.COM (short username oryx)
I20260620 23:05:46.783694 18993 webserver.cc:460] Webserver started at http://127.12.2.124:33591/ using document root <none> and password file <none>
I20260620 23:05:46.783946 18993 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:46.784031 18993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:46.786367 18993 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 23:05:46.787849 19200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:46.788216 18993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:46.788318 18993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/wal
uuid: "3f9a7aaaa9174ba6a11acf9635519a8d"
format_stamp: "Formatted at 2026-06-20 23:05:46 on dist-test-slave-7g8g"
server_key: "6924fd51fbc9a4273267d701fa14d3d0"
server_key_iv: "ce99f7a7a5cdd3786d948beabc23a1fc"
server_key_version: "encryptionkey@0"
I20260620 23:05:46.788501 18993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:46.797063 18993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:46.800172 18993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:46.845674 18993 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.124:33779
I20260620 23:05:46.845726 19251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.124:33779 every 8 connection(s)
I20260620 23:05:46.846633 18993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
I20260620 23:05:46.851475 19252 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:05:46.855043 19252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: Bootstrap starting.
I20260620 23:05:46.878394 19252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:05:46.878999 19252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: Bootstrap complete.
I20260620 23:05:46.880089 19252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } }
I20260620 23:05:46.880417 19252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f9a7aaaa9174ba6a11acf9635519a8d, State: Initialized, Role: LEARNER
I20260620 23:05:46.880681 19252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } }
I20260620 23:05:46.882506 19255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:46.882792 19252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:46.884444 19255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: This master's current role is: LEARNER
I20260620 23:05:46.891078 19266 catalog_manager.cc:1259] Loaded cluster ID: 1befd225d77342798bc52be4cecf56e2
I20260620 23:05:46.891175 19266 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: loading cluster ID for follower catalog manager: success
I20260620 23:05:46.894421 19266 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: acquiring CA information for follower catalog manager: success
I20260620 23:05:46.896680 19266 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:05:47.100425 19189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:47.100718 19189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:47.100863 19189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:47.108294 19189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:47.108393 19189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:47.108542 19189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:47.117921 19189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab
--principal=oryx/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:39893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=40137
--tserver_master_addrs=127.12.2.126:41439,127.12.2.125:42515,127.12.2.124:33779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:47.119210 19189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:47.120718 19189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:47.133622 19271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:47.133591 19274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:47.133651 19272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:47.134225 19189 server_base.cc:1034] running on GCE node
I20260620 23:05:47.134979 19189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:47.135879 19189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:47.137070 19189 hybrid_clock.cc:648] HybridClock initialized: now 1781996747137020 us; error 47 us; skew 500 ppm
I20260620 23:05:47.137389 19189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:47 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996747, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:47.140434 19189 init.cc:377] Logged in from keytab as oryx/127.12.2.65@KRBTEST.COM (short username oryx)
I20260620 23:05:47.141831 19189 webserver.cc:460] Webserver started at http://127.12.2.65:40137/ using document root <none> and password file <none>
I20260620 23:05:47.142506 19189 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:47.142643 19189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:47.148211 19189 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 23:05:47.150717 19281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:47.152063 19189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:05:47.152215 19189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "c5bae359efc64d17b9591adc8dc7b185"
format_stamp: "Formatted at 2026-06-20 23:05:44 on dist-test-slave-7g8g"
server_key: "57fc910766059ec6f6b6a24da23cceeb"
server_key_iv: "067db55f5bd6178372149e963ca00b19"
server_key_version: "encryptionkey@0"
I20260620 23:05:47.153153 19189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:47.171309 19189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:47.175264 19189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:47.175539 19189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:47.176887 19189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:47.178191 19189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:47.178272 19189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:47.178354 19189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:47.178411 19189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:47.210433 19189 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:39893
I20260620 23:05:47.210489 19393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:39893 every 8 connection(s)
I20260620 23:05:47.211931 19189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:47.222187 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 19189
I20260620 23:05:47.222570 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 18627
Jun 20 23:05:47 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996747, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
Jun 20 23:05:47 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996747, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.124@KRBTEST.COM
I20260620 23:05:47.265256 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.214579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:38519 (local address 127.12.2.126:41439)
0620 23:05:47.214784 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:47.214791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:47.231519 (+ 16728us) server_negotiation.cc:407] Connection header received
0620 23:05:47.235935 (+ 4416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:47.235943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:47.236032 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:47.236101 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:47.238721 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.239751 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.247049 (+ 7298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.247712 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.247809 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.258243 (+ 10434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:47.258252 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:47.258256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:47.258298 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:47.262357 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.263195 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.263201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.263205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.263337 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.263767 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.263772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.263776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.264109 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:47.264246 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:47.264825 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 23:05:47.265010 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 23:05:47 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996747, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.125@KRBTEST.COM
Jun 20 23:05:47 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.2.66@KRBTEST.COM
I20260620 23:05:47.266355 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.215223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.65:38519)
0620 23:05:47.231398 (+ 16175us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:47.231417 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:47.235568 (+ 4151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:47.236667 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:47.236680 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:47.237653 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:47.238524 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.238547 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.239935 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.239940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.242022 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.242037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.248413 (+ 6376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.248420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.248712 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.250013 (+ 1301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:47.250050 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:47.257683 (+ 7633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:47.262508 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.262513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.262516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.263040 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.263474 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.263478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.263481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.263634 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.264385 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:47.264389 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:47.264527 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 23:05:47.265538 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0620 23:05:47.265796 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16027,"mutex_wait_us":187,"thread_start_us":13222,"threads_started":1}
I20260620 23:05:47.266858 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.215625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:50871 (local address 127.12.2.124:33779)
0620 23:05:47.242640 (+ 27015us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:47.242646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:47.242662 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:05:47.242719 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:47.242725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:47.242803 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:47.242900 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:47.244214 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.245649 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.252022 (+ 6373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.252584 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.252825 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.257840 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:47.257847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:47.257853 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:47.257892 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:47.261281 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.262216 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.262221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.262225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.262370 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.262959 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.262964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.262968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.263307 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:47.263455 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:47.266300 (+ 2845us) server_negotiation.cc:299] Negotiation successful
0620 23:05:47.266512 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26855,"thread_start_us":26714,"threads_started":1}
I20260620 23:05:47.267127 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.215216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:33779 (local address 127.12.2.65:50871)
0620 23:05:47.229166 (+ 13950us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:47.229224 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:47.242150 (+ 12926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:47.243166 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:47.243171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:47.243757 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:47.244026 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.244037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.245817 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.245822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.249593 (+ 3771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.249607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.253073 (+ 3466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.253082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.253293 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.254006 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:47.254027 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:47.257554 (+ 3527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:47.261439 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.261447 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.261466 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.262059 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.262498 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.262502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.262506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.262802 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.263529 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:47.263537 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:47.264005 (+ 468us) client_negotiation.cc:769] Sending connection context
0620 23:05:47.264331 (+ 326us) client_negotiation.cc:240] Negotiation successful
0620 23:05:47.264886 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13583,"thread_start_us":15727,"threads_started":1}
I20260620 23:05:47.268206 19395 heartbeater.cc:347] Connected to a master server at 127.12.2.126:41439
I20260620 23:05:47.268489 19394 heartbeater.cc:347] Connected to a master server at 127.12.2.124:33779
I20260620 23:05:47.269047 19395 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:47.269052 19394 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:47.270032 19395 heartbeater.cc:511] Master 127.12.2.126:41439 requested a full tablet report, sending...
I20260620 23:05:47.270041 19394 heartbeater.cc:511] Master 127.12.2.124:33779 requested a full tablet report, sending...
I20260620 23:05:47.271934 18432 ts_manager.cc:194] Re-registered known tserver with Master: c5bae359efc64d17b9591adc8dc7b185 (127.12.2.65:39893)
I20260620 23:05:47.273018 18432 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:38519
I20260620 23:05:47.273283 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.231498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:36697 (local address 127.12.2.125:42515)
0620 23:05:47.251942 (+ 20444us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:47.251947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:47.256907 (+ 4960us) server_negotiation.cc:407] Connection header received
0620 23:05:47.257047 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:47.257052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:47.257130 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:47.257226 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:47.258527 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.259634 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.261763 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.262381 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.262470 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.267442 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:47.267449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:47.267453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:47.267494 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:47.270786 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.271520 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.271525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.271529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.271645 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.271990 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.271995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.271999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.272280 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:47.272402 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:47.272744 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 23:05:47.272929 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20331,"thread_start_us":20253,"threads_started":1}
I20260620 23:05:47.273576 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.231063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42515 (local address 127.12.2.65:36697)
0620 23:05:47.256797 (+ 25734us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:47.256813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:47.256937 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:47.257436 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:47.257440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:47.258038 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:47.258334 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.258345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.259821 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.259826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.261481 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.261497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.262746 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.262752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.262954 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.263443 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:47.263464 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:47.266816 (+ 3352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:47.270963 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.270968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.270972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.271408 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.271748 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.271753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.271757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.271884 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.272388 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:47.272393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:47.272510 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 23:05:47.272614 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:05:47.272897 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25586,"thread_start_us":25421,"threads_started":1}
I20260620 23:05:47.274390 19396 heartbeater.cc:347] Connected to a master server at 127.12.2.125:42515
I20260620 23:05:47.274507 19396 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:47.274616 19217 ts_manager.cc:194] Registered new tserver with Master: c5bae359efc64d17b9591adc8dc7b185 (127.12.2.65:39893)
I20260620 23:05:47.274777 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.223871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.12.2.66:46755)
0620 23:05:47.250511 (+ 26640us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:47.250520 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:47.251521 (+ 1001us) server_negotiation.cc:407] Connection header received
0620 23:05:47.251612 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:47.251620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:47.251927 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:47.252077 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:47.254231 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.256961 (+ 2730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.258768 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.259643 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.259769 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.267186 (+ 7417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:47.267203 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:47.267219 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:47.267262 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:47.270889 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.271797 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.271805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.271809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.271954 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.272393 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.272398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.272400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.272785 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:47.273046 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:47.274143 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0620 23:05:47.274389 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26452,"thread_start_us":26653,"threads_started":1}
I20260620 23:05:47.274956 19396 heartbeater.cc:511] Master 127.12.2.125:42515 requested a full tablet report, sending...
I20260620 23:05:47.276240 18750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:47.276602 19030 ts_manager.cc:194] Registered new tserver with Master: c5bae359efc64d17b9591adc8dc7b185 (127.12.2.65:39893)
I20260620 23:05:47.285826 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.278813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:33779 (local address 127.0.0.1:49858)
0620 23:05:47.279291 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:47.279305 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:47.279406 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:47.279738 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:47.279740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:47.279757 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:47.279889 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.279896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.281304 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.281310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.283176 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.283193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.284732 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.284738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.284904 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.284935 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:05:47.285065 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 23:05:47.285162 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":248,"threads_started":1}
I20260620 23:05:47.286288 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.278919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.12.2.124:33779)
0620 23:05:47.279125 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:47.279130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:47.279370 (+ 240us) server_negotiation.cc:407] Connection header received
0620 23:05:47.279509 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:47.279514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:47.279621 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:47.279735 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:47.279993 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.281096 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.283331 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.284562 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.284733 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.284961 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:47.286014 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0620 23:05:47.286115 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:05:47.286921 19227 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 23:05:47.287480 19255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a99eb094dc4ae295ad268a70895388. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:47.287767 19255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: This master's current role is: LEARNER
I20260620 23:05:47.364825 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 18627
I20260620 23:05:47.378005 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:46755
--local_ip_for_outbound_sockets=127.12.2.66
--tserver_master_addrs=127.12.2.126:41439
--webserver_port=37235
--webserver_interface=127.12.2.66
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab
--principal=oryx/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.2.126:41439,127.12.2.125:42515
--tserver_master_addrs=127.12.2.126:41439,127.12.2.125:42515,127.12.2.124:33779 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:47.688968 19407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:47.689261 19407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:47.689404 19407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:47.696683 19407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:47.696795 19407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:47.696939 19407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:47.706574 19407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab
--principal=oryx/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:46755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=37235
--tserver_master_addrs=127.12.2.126:41439,127.12.2.125:42515,127.12.2.124:33779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:47.707886 19407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:47.709223 19407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:47.721635 19413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:47.721660 19415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:47.721876 19412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:47.722474 19407 server_base.cc:1034] running on GCE node
I20260620 23:05:47.723112 19407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:47.724014 19407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:47.725205 19407 hybrid_clock.cc:648] HybridClock initialized: now 1781996747725156 us; error 41 us; skew 500 ppm
I20260620 23:05:47.725517 19407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:47 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996747, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:47.728969 19407 init.cc:377] Logged in from keytab as oryx/127.12.2.66@KRBTEST.COM (short username oryx)
I20260620 23:05:47.730425 19407 webserver.cc:460] Webserver started at http://127.12.2.66:37235/ using document root <none> and password file <none>
I20260620 23:05:47.731122 19407 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:47.731243 19407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:47.736452 19407 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:05:47.739112 19422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:47.740396 19407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:47.740530 19407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "174c8d98e2bc45fcb171db3de048efbf"
format_stamp: "Formatted at 2026-06-20 23:05:44 on dist-test-slave-7g8g"
server_key: "4e171696d5b6490acb6a1f5293cb3552"
server_key_iv: "3f55c496fbeb06a552a70e0766c4b5af"
server_key_version: "encryptionkey@0"
I20260620 23:05:47.741449 19407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:47.766966 19407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:47.771068 19407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:47.771292 19407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:47.772149 19407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:47.773422 19407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:47.773509 19407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:47.773615 19407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:47.773689 19407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:47.803764 19407 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:46755
I20260620 23:05:47.803794 19534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:46755 every 8 connection(s)
I20260620 23:05:47.805146 19407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:47.806672 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 19407
I20260620 23:05:47.806993 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 18760
Jun 20 23:05:47 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996747, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.125@KRBTEST.COM
Jun 20 23:05:47 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.2.67@KRBTEST.COM
Jun 20 23:05:47 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996747, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.124@KRBTEST.COM
I20260620 23:05:47.860651 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.808530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.12.2.67:41215)
0620 23:05:47.827869 (+ 19339us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:47.827879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:47.827906 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:05:47.827965 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:47.827972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:47.828228 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:47.828383 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:47.830355 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.831891 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.833672 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.834483 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.834582 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.847987 (+ 13405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:47.848005 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:47.848022 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:47.848070 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:47.851495 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.852312 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.852320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.852324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.852468 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.857648 (+ 5180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.857654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.857657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.858084 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:47.858350 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:47.860009 (+ 1659us) server_negotiation.cc:299] Negotiation successful
0620 23:05:47.860285 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19175,"thread_start_us":23617,"threads_started":1}
I20260620 23:05:47.861796 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.808842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:39675 (local address 127.12.2.125:42515)
0620 23:05:47.827454 (+ 18612us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:47.827459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:47.829077 (+ 1618us) server_negotiation.cc:407] Connection header received
0620 23:05:47.829299 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:47.829305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:47.829380 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:47.829454 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:47.830962 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.831964 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.834416 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.835023 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.835109 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.839617 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:47.839624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:47.839628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:47.839680 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:47.842815 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.859130 (+ 16315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.859135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.859139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.859296 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.859745 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.859749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.859753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.860111 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:47.860232 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:47.861225 (+ 993us) server_negotiation.cc:299] Negotiation successful
0620 23:05:47.861447 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18488,"thread_start_us":18388,"threads_started":1}
I20260620 23:05:47.863659 19175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: NON_VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: true } } }
I20260620 23:05:47.864043 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.808734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42515 (local address 127.12.2.66:39675)
0620 23:05:47.828981 (+ 20247us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:47.829000 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:47.829164 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:47.829693 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:47.829697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:47.830275 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:47.830749 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.830774 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.832135 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.832139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.834263 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.834278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.835435 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.835442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.835759 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.836383 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:47.836413 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:47.839350 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:47.843003 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.843012 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.843029 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.858982 (+ 15953us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.859441 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.859445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.859448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.859598 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.860329 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:47.860338 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:47.860764 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 23:05:47.862066 (+ 1302us) client_negotiation.cc:240] Negotiation successful
0620 23:05:47.862603 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20126,"spinlock_wait_cycles":56704,"thread_start_us":14908,"threads_started":1}
I20260620 23:05:47.864686 19175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:47.864972 18883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:47.865018 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.807998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:58801 (local address 127.12.2.124:33779)
0620 23:05:47.818219 (+ 10221us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:47.818226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:47.824256 (+ 6030us) server_negotiation.cc:407] Connection header received
0620 23:05:47.826401 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:47.826407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:47.826486 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:47.826577 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:47.835773 (+ 9196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.837014 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.842800 (+ 5786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.843401 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.843500 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.852677 (+ 9177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:47.852684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:47.852689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:47.852727 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:47.856486 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.858433 (+ 1947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.858439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.858443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.858575 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.861506 (+ 2931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.861511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.861515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.861828 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:47.862072 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:47.864421 (+ 2349us) server_negotiation.cc:299] Negotiation successful
0620 23:05:47.864684 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10084,"thread_start_us":10005,"threads_started":1}
I20260620 23:05:47.866233 19537 heartbeater.cc:347] Connected to a master server at 127.12.2.125:42515
I20260620 23:05:47.866570 19537 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:47.867419 19537 heartbeater.cc:511] Master 127.12.2.125:42515 requested a full tablet report, sending...
I20260620 23:05:47.868398 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.808645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:33779 (local address 127.12.2.66:58801)
0620 23:05:47.824089 (+ 15444us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:47.824145 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:47.826122 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:47.826861 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:47.826870 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:47.827709 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:47.835602 (+ 7893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.835614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.837202 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.837207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.838704 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.838717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.843839 (+ 5122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.843845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.844009 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.845297 (+ 1288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:47.845318 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:47.847712 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:47.856652 (+ 8940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.856656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.856660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.857211 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.858723 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.858728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.858732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.858995 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.862142 (+ 3147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:47.862145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:47.862285 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 23:05:47.865237 (+ 2952us) client_negotiation.cc:240] Negotiation successful
0620 23:05:47.865518 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15141,"thread_start_us":24550,"threads_started":1}
I20260620 23:05:47.869095 19535 heartbeater.cc:347] Connected to a master server at 127.12.2.124:33779
I20260620 23:05:47.869105 19030 ts_manager.cc:194] Registered new tserver with Master: 174c8d98e2bc45fcb171db3de048efbf (127.12.2.66:46755)
I20260620 23:05:47.869236 19535 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:47.869763 19535 heartbeater.cc:511] Master 127.12.2.124:33779 requested a full tablet report, sending...
I20260620 23:05:47.871433 19217 ts_manager.cc:194] Registered new tserver with Master: 174c8d98e2bc45fcb171db3de048efbf (127.12.2.66:46755)
Jun 20 23:05:47 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996747, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:47.882776 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.827606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.66:46317)
0620 23:05:47.861261 (+ 33655us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:47.861279 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:47.861384 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:47.865663 (+ 4279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:47.865668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:47.866241 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:47.866514 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.866526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.869376 (+ 2850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.869381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.871053 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:47.871069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.872322 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.872330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:47.872509 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.873188 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:47.873211 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:47.876082 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:47.880198 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.880203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.880206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.880644 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.881065 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:47.881070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:47.881074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:47.881242 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:47.881876 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:47.881881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:47.882024 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 23:05:47.882147 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 23:05:47.882412 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33522,"thread_start_us":33382,"threads_started":1}
I20260620 23:05:47.883783 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:47.827575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:46317 (local address 127.12.2.126:41439)
0620 23:05:47.859416 (+ 31841us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:47.859422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:47.862423 (+ 3001us) server_negotiation.cc:407] Connection header received
0620 23:05:47.862483 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:47.862489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:47.862563 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:47.862656 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:47.866689 (+ 4033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.867836 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.871241 (+ 3405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:47.871939 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:47.872053 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:47.876360 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:47.876368 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:47.876373 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:47.876414 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:47.879986 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.880781 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.880788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.880792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.880944 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:47.881356 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:47.881361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:47.881365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:47.881692 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:47.881821 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:47.883134 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0620 23:05:47.883408 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31718,"thread_start_us":31609,"threads_started":1}
I20260620 23:05:47.884799 19536 heartbeater.cc:347] Connected to a master server at 127.12.2.126:41439
I20260620 23:05:47.884940 19536 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:47.885352 19536 heartbeater.cc:511] Master 127.12.2.126:41439 requested a full tablet report, sending...
I20260620 23:05:47.886371 18432 ts_manager.cc:194] Re-registered known tserver with Master: 174c8d98e2bc45fcb171db3de048efbf (127.12.2.66:46755)
I20260620 23:05:47.887564 18432 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:46317
I20260620 23:05:47.887647 19547 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388: attempting to promote NON_VOTER 3f9a7aaaa9174ba6a11acf9635519a8d to VOTER
I20260620 23:05:47.888283 19547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } }
I20260620 23:05:47.890354 19227 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 LEARNER]: Refusing update from remote peer 07a99eb094dc4ae295ad268a70895388: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 23:05:47.890358 19040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 FOLLOWER]: Refusing update from remote peer 07a99eb094dc4ae295ad268a70895388: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 23:05:47.890877 19548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 23:05:47.891548 19548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 23:05:47.894217 19175 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3f9a7aaaa9174ba6a11acf9635519a8d (127.12.2.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:47.894855 19040 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3f9a7aaaa9174ba6a11acf9635519a8d (127.12.2.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:47.897897 19547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:47.898139 19547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:47.897982 19548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:47.898258 19548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:47.900198 19555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:47.900527 19555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:05:47.900789 19547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:47.901029 19547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:47.901340 19227 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3f9a7aaaa9174ba6a11acf9635519a8d (127.12.2.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:47.904399 19556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07a99eb094dc4ae295ad268a70895388" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:47.904707 19556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:05:47.964383 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 18760
I20260620 23:05:47.976812 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:41215
--local_ip_for_outbound_sockets=127.12.2.67
--tserver_master_addrs=127.12.2.126:41439
--webserver_port=43639
--webserver_interface=127.12.2.67
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab
--principal=oryx/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.2.126:41439,127.12.2.125:42515
--tserver_master_addrs=127.12.2.126:41439,127.12.2.125:42515,127.12.2.124:33779 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:05:48.278162 19395 heartbeater.cc:503] Master 127.12.2.126:41439 was elected leader, sending a full tablet report...
W20260620 23:05:48.291528 19557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:48.291814 19557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:48.291949 19557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:48.299276 19557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:48.299402 19557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:48.299551 19557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:48.309140 19557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab
--principal=oryx/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:41215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=43639
--tserver_master_addrs=127.12.2.126:41439,127.12.2.125:42515,127.12.2.124:33779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:48.310459 19557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:48.311795 19557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:48.325162 19563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:48.325232 19565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:48.325201 19562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:48.326408 19557 server_base.cc:1034] running on GCE node
I20260620 23:05:48.327111 19557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:48.328059 19557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:48.329270 19557 hybrid_clock.cc:648] HybridClock initialized: now 1781996748329211 us; error 47 us; skew 500 ppm
I20260620 23:05:48.329572 19557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:48 dist-test-slave-7g8g krb5kdc[18387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996748, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:48.332645 19557 init.cc:377] Logged in from keytab as oryx/127.12.2.67@KRBTEST.COM (short username oryx)
I20260620 23:05:48.334076 19557 webserver.cc:460] Webserver started at http://127.12.2.67:43639/ using document root <none> and password file <none>
I20260620 23:05:48.334710 19557 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:48.334823 19557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:48.340046 19557 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:05:48.342794 19572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:48.344182 19557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:05:48.344323 19557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "f8641366ac034f14a249b162e30a5703"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-7g8g"
server_key: "bedb7de7fbfd4c28b4138af2421efde7"
server_key_iv: "7c9b61c564d25f6a65fea0d77ba42942"
server_key_version: "encryptionkey@0"
I20260620 23:05:48.345331 19557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:48.371381 19557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:48.375667 19557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:48.375942 19557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:48.376866 19557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:48.378098 19557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:48.378177 19557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:48.378258 19557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:48.378310 19557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:48.408196 19557 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:41215
I20260620 23:05:48.408264 19684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:41215 every 8 connection(s)
I20260620 23:05:48.409567 19557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:48.416404 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 19557
Jun 20 23:05:48 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996748, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.125@KRBTEST.COM
Jun 20 23:05:48 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996748, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.124@KRBTEST.COM
I20260620 23:05:48.448905 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.412609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:50777 (local address 127.12.2.125:42515)
0620 23:05:48.413015 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.413021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.425762 (+ 12741us) server_negotiation.cc:407] Connection header received
0620 23:05:48.429426 (+ 3664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.429432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.429524 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.429636 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:48.431466 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.432592 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.434526 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.435248 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.435372 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.440441 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:48.440447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:48.440452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:48.440492 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:48.444296 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.445678 (+ 1382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.445686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.445690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.445858 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.446409 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.446414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.446418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.446805 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:48.446958 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.448286 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.448495 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":794,"threads_started":1}
I20260620 23:05:48.451709 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.413398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42515 (local address 127.12.2.67:50777)
0620 23:05:48.425536 (+ 12138us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:48.425603 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:48.428617 (+ 3014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:48.429913 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:48.429923 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:48.430500 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:48.431290 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.431310 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.432745 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.432751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.434365 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.434380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.435470 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.435477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.435779 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.436560 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:48.436591 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:48.440162 (+ 3571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:48.444723 (+ 4561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:48.444739 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:48.444763 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:48.445507 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:48.445991 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:48.445996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:48.446000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:48.446278 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:48.447029 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:48.447039 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:48.447496 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 23:05:48.447800 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 23:05:48.449831 (+ 2031us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11811,"thread_start_us":11596,"threads_started":1}
Jun 20 23:05:48 dist-test-slave-7g8g krb5kdc[18387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996748, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:48.453423 19687 heartbeater.cc:347] Connected to a master server at 127.12.2.125:42515
I20260620 23:05:48.453738 19687 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:48.454644 19687 heartbeater.cc:511] Master 127.12.2.125:42515 requested a full tablet report, sending...
I20260620 23:05:48.456518 19030 ts_manager.cc:194] Registered new tserver with Master: f8641366ac034f14a249b162e30a5703 (127.12.2.67:41215)
I20260620 23:05:48.461333 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.412404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:52689 (local address 127.12.2.126:41439)
0620 23:05:48.424878 (+ 12474us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.424886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.434525 (+ 9639us) server_negotiation.cc:407] Connection header received
0620 23:05:48.434742 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.434748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.434834 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.434901 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:48.437108 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.438666 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.447978 (+ 9312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.448691 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.448804 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.454568 (+ 5764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:48.454576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:48.454581 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:48.454624 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:48.458044 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.458892 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.458898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.458901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.459109 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.459478 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.459484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.459487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.459853 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:48.459976 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.460737 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.460960 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12297,"thread_start_us":13389,"threads_started":1}
I20260620 23:05:48.461777 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.412328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:35535 (local address 127.12.2.124:33779)
0620 23:05:48.429283 (+ 16955us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.429287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.432135 (+ 2848us) server_negotiation.cc:407] Connection header received
0620 23:05:48.432263 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.432269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.432354 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.432470 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:48.433900 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.434980 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.436966 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.437589 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.437699 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.449403 (+ 11704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:48.449410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:48.449414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:48.449453 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:48.457630 (+ 8177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.458569 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.458575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.458579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.458739 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.459980 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.459986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.459989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.460402 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:48.460536 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.461211 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.461422 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16838,"thread_start_us":16730,"threads_started":1}
I20260620 23:05:48.461797 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.413879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.67:52689)
0620 23:05:48.432612 (+ 18733us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:48.432628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:48.434613 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:48.435127 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:48.435131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:48.436544 (+ 1413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:48.436956 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.436966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.438875 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.438880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.446956 (+ 8076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.446973 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.449122 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.449129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.449330 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.450240 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:48.450263 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:48.454360 (+ 4097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:48.458136 (+ 3776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:48.458140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:48.458144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:48.458664 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:48.459195 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:48.459201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:48.459205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:48.459356 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:48.460177 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:48.460183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:48.460355 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 23:05:48.460490 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 23:05:48.461388 (+ 898us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18624,"mutex_wait_us":4,"thread_start_us":17884,"threads_started":1}
I20260620 23:05:48.462519 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.413373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:33779 (local address 127.12.2.67:35535)
0620 23:05:48.432029 (+ 18656us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:48.432048 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:48.432164 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:48.432657 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:48.432663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:48.433391 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:48.433723 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.433735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.435138 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.435143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.436754 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.436767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.437976 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.437982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.438147 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.445013 (+ 6866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:48.445037 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:48.448969 (+ 3932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:48.457787 (+ 8818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:48.457793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:48.457797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:48.458454 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:48.459705 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:48.459710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:48.459713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:48.459846 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:48.460595 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:48.460600 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:48.460784 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 23:05:48.460905 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 23:05:48.462136 (+ 1231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18491,"mutex_wait_us":1071,"spinlock_wait_cycles":72960,"thread_start_us":18828,"threads_started":1}
I20260620 23:05:48.463660 19685 heartbeater.cc:347] Connected to a master server at 127.12.2.124:33779
I20260620 23:05:48.463754 19686 heartbeater.cc:347] Connected to a master server at 127.12.2.126:41439
I20260620 23:05:48.463807 19685 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:48.463893 19686 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:48.464377 19685 heartbeater.cc:511] Master 127.12.2.124:33779 requested a full tablet report, sending...
I20260620 23:05:48.464424 19686 heartbeater.cc:511] Master 127.12.2.126:41439 requested a full tablet report, sending...
I20260620 23:05:48.465570 19217 ts_manager.cc:194] Registered new tserver with Master: f8641366ac034f14a249b162e30a5703 (127.12.2.67:41215)
I20260620 23:05:48.465814 18432 ts_manager.cc:194] Re-registered known tserver with Master: f8641366ac034f14a249b162e30a5703 (127.12.2.67:41215)
I20260620 23:05:48.466902 18432 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:52689
I20260620 23:05:48.478626 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.440526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.12.2.125:42515)
0620 23:05:48.449141 (+ 8615us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.449146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.458149 (+ 9003us) server_negotiation.cc:407] Connection header received
0620 23:05:48.458192 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.458197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.458263 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.458341 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:48.463069 (+ 4728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.464754 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.468590 (+ 3836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.469182 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.469304 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.472778 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:48.472784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:48.472789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:48.472827 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:48.475940 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.476707 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.476713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.476717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.476836 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.477218 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.477222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.477226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.477566 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:48.477833 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.478083 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.478282 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8511,"thread_start_us":16345,"threads_started":1}
I20260620 23:05:48.481053 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.439973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.12.2.126:41439)
0620 23:05:48.462205 (+ 22232us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.462211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.462230 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:05:48.462276 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.462281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.462346 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.462438 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:48.467432 (+ 4994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.468440 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.471341 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.471794 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.471908 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.474732 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:48.474737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:48.474740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:48.474770 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:48.477614 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.478961 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.478967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.478971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.479091 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.479536 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.479540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.479542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.479800 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:48.479994 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.480594 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.480819 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21798,"thread_start_us":19660,"threads_started":1}
I20260620 23:05:48.481606 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.438383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.12.2.124:33779)
0620 23:05:48.461980 (+ 23597us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.461987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.462003 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:05:48.462048 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.462052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.462121 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.462229 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:48.467568 (+ 5339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.468674 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.470874 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.471543 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.471669 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.474791 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:48.474798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:48.474803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:48.474841 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:48.478182 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.479008 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.479013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.479016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.479152 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.479508 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.479513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.479517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.479862 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:48.480223 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.481122 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.481329 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23355,"thread_start_us":23520,"threads_started":1}
I20260620 23:05:48.484946 18432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:48.491628 18432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:48.525098 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.511384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.12.2.65:39893)
0620 23:05:48.511965 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.511975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.518148 (+ 6173us) server_negotiation.cc:407] Connection header received
0620 23:05:48.518440 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.518450 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.518766 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.518975 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:48.519798 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.521069 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.523170 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.524095 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.524205 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.524418 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.524678 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.524843 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":1019,"threads_started":1}
I20260620 23:05:48.525063 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.511105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39893 (local address 127.0.0.1:40054)
0620 23:05:48.518028 (+ 6923us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:48.518046 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:48.518249 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:48.519241 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:48.519246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:48.519272 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:48.519610 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.519621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.521199 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.521205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.523042 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.523054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.524302 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.524310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.524502 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.524527 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 23:05:48.524598 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 23:05:48.524721 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6803,"thread_start_us":8752,"threads_started":1}
I20260620 23:05:48.528007 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.511781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:41215 (local address 127.0.0.1:33308)
0620 23:05:48.520069 (+ 8288us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:48.520082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:48.520179 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:48.521308 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:48.521312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:48.521332 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:48.521801 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.521815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.523630 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.523635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.524938 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.524949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.527093 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.527100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.527398 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.527431 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:05:48.527529 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 23:05:48.527647 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8179,"mutex_wait_us":1,"thread_start_us":6551,"threads_started":1}
I20260620 23:05:48.528074 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.512202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.12.2.67:41215)
0620 23:05:48.512896 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.512905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.520182 (+ 7277us) server_negotiation.cc:407] Connection header received
0620 23:05:48.520299 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.520307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.520545 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.520731 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:48.521981 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.523523 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.525067 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.526421 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.526538 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.526798 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.527615 (+ 817us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.527774 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":588,"threads_started":1}
I20260620 23:05:48.529966 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.510457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:46755 (local address 127.0.0.1:41878)
0620 23:05:48.516754 (+ 6297us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:48.516777 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:48.517466 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:48.521097 (+ 3631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:48.521101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:48.521124 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:48.521584 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.521595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.525121 (+ 3526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.525126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.527582 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.527595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.529269 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.529275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.529434 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.529465 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:05:48.529540 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:05:48.529655 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":316,"threads_started":1}
I20260620 23:05:48.530005 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.510669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.12.2.66:46755)
0620 23:05:48.520590 (+ 9921us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.520598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.520622 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:05:48.520717 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.520724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.520923 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.521097 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:48.521981 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.523515 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.527755 (+ 4240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.529090 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.529200 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.529401 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.529597 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.529762 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9768,"thread_start_us":9681,"threads_started":1}
I20260620 23:05:48.531152 19329 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3145561ba344529e87b000f0a83e3e (DEFAULT_TABLE table=test-table [id=3386ad840dea4a75bf937481baf0cb1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:48.531574 19620 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3145561ba344529e87b000f0a83e3e (DEFAULT_TABLE table=test-table [id=3386ad840dea4a75bf937481baf0cb1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:48.532665 19329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3145561ba344529e87b000f0a83e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:48.532873 19620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3145561ba344529e87b000f0a83e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:48.533087 19470 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3145561ba344529e87b000f0a83e3e (DEFAULT_TABLE table=test-table [id=3386ad840dea4a75bf937481baf0cb1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:48.534089 19470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3145561ba344529e87b000f0a83e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:48.545509 19719 tablet_bootstrap.cc:492] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf: Bootstrap starting.
I20260620 23:05:48.547381 19720 tablet_bootstrap.cc:492] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: Bootstrap starting.
I20260620 23:05:48.549026 19721 tablet_bootstrap.cc:492] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: Bootstrap starting.
I20260620 23:05:48.550765 19720 tablet_bootstrap.cc:654] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:48.550765 19719 tablet_bootstrap.cc:654] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:48.551863 19721 tablet_bootstrap.cc:654] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:48.551963 19720 log.cc:824] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:48.551963 19719 log.cc:824] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:48.553483 19721 log.cc:824] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:48.555210 19719 tablet_bootstrap.cc:492] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf: No bootstrap required, opened a new log
I20260620 23:05:48.555466 19719 ts_tablet_manager.cc:1397] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 23:05:48.556510 19721 tablet_bootstrap.cc:492] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: No bootstrap required, opened a new log
I20260620 23:05:48.556721 19721 ts_tablet_manager.cc:1397] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 23:05:48.560328 19720 tablet_bootstrap.cc:492] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: No bootstrap required, opened a new log
I20260620 23:05:48.560601 19720 ts_tablet_manager.cc:1397] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260620 23:05:48.562964 19719 raft_consensus.cc:348] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } } peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } }
I20260620 23:05:48.563385 19719 raft_consensus.cc:374] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:48.563472 19719 raft_consensus.cc:729] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174c8d98e2bc45fcb171db3de048efbf, State: Initialized, Role: FOLLOWER
I20260620 23:05:48.564028 19719 consensus_queue.cc:260] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } } peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } }
I20260620 23:05:48.565764 19719 ts_tablet_manager.cc:1428] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260620 23:05:48.565652 19721 raft_consensus.cc:348] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } } peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } }
I20260620 23:05:48.566012 19536 heartbeater.cc:503] Master 127.12.2.126:41439 was elected leader, sending a full tablet report...
I20260620 23:05:48.566067 19721 raft_consensus.cc:374] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:48.566159 19721 raft_consensus.cc:729] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8641366ac034f14a249b162e30a5703, State: Initialized, Role: FOLLOWER
I20260620 23:05:48.566751 19721 consensus_queue.cc:260] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } } peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } }
I20260620 23:05:48.572468 19686 heartbeater.cc:503] Master 127.12.2.126:41439 was elected leader, sending a full tablet report...
I20260620 23:05:48.573612 19721 ts_tablet_manager.cc:1428] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: Time spent starting tablet: real 0.017s user 0.008s sys 0.009s
I20260620 23:05:48.574537 19720 raft_consensus.cc:348] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } } peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } }
I20260620 23:05:48.575322 19720 raft_consensus.cc:374] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:48.575474 19720 raft_consensus.cc:729] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5bae359efc64d17b9591adc8dc7b185, State: Initialized, Role: FOLLOWER
I20260620 23:05:48.576089 19720 consensus_queue.cc:260] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } } peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } }
I20260620 23:05:48.577893 19720 ts_tablet_manager.cc:1428] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260620 23:05:48.662379 19688 tablet.cc:2369] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:48.689731 19725 raft_consensus.cc:482] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:48.690017 19725 raft_consensus.cc:504] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } } peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } }
I20260620 23:05:48.691682 19725 leader_election.cc:288] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5bae359efc64d17b9591adc8dc7b185 (127.12.2.65:39893), f8641366ac034f14a249b162e30a5703 (127.12.2.67:41215)
I20260620 23:05:48.698249 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.692012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:51531 (local address 127.12.2.65:39893)
0620 23:05:48.692207 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.692213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.692890 (+ 677us) server_negotiation.cc:407] Connection header received
0620 23:05:48.693047 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.693050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.693105 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.693172 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:48.694089 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.694793 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.696660 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.697417 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.697526 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.697620 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.697893 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.698009 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:05:48.698278 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.692207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:38751 (local address 127.12.2.67:41215)
0620 23:05:48.692412 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.692418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.692836 (+ 418us) server_negotiation.cc:407] Connection header received
0620 23:05:48.693026 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.693030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.693088 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.693160 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:48.694054 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.694785 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.696654 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.697412 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.697527 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.697629 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.697931 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.698054 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:05:48.698438 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.691947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:41215 (local address 127.12.2.66:38751)
0620 23:05:48.692728 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:48.692743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:48.692897 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:48.693183 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:48.693187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:48.693208 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:48.693917 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.693927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.694921 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.694948 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.696511 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.696522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.697667 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.697674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.697785 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.697808 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:05:48.697895 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:05:48.698016 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":2,"thread_start_us":468,"threads_started":1}
I20260620 23:05:48.698781 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.692019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39893 (local address 127.12.2.66:51531)
0620 23:05:48.692789 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:48.692803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:48.692921 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:48.693191 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:48.693195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:48.693245 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:48.693948 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.693957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.694910 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.694915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.696514 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:48.696524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.697615 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.697621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:48.697750 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.697776 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:05:48.697855 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:05:48.697984 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":53,"spinlock_wait_cycles":2432,"thread_start_us":438,"threads_started":1}
I20260620 23:05:48.699404 19640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3145561ba344529e87b000f0a83e3e" candidate_uuid: "174c8d98e2bc45fcb171db3de048efbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8641366ac034f14a249b162e30a5703" is_pre_election: true
I20260620 23:05:48.699607 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3145561ba344529e87b000f0a83e3e" candidate_uuid: "174c8d98e2bc45fcb171db3de048efbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5bae359efc64d17b9591adc8dc7b185" is_pre_election: true
I20260620 23:05:48.699870 19640 raft_consensus.cc:2454] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174c8d98e2bc45fcb171db3de048efbf in term 0.
I20260620 23:05:48.700029 19349 raft_consensus.cc:2454] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174c8d98e2bc45fcb171db3de048efbf in term 0.
I20260620 23:05:48.700453 19426 leader_election.cc:302] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174c8d98e2bc45fcb171db3de048efbf, f8641366ac034f14a249b162e30a5703; no voters:
I20260620 23:05:48.700811 19725 raft_consensus.cc:2781] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:48.700946 19725 raft_consensus.cc:482] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:48.701040 19725 raft_consensus.cc:3037] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:48.703116 19725 raft_consensus.cc:504] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } } peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } }
I20260620 23:05:48.703622 19725 leader_election.cc:288] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [CANDIDATE]: Term 1 election: Requested vote from peers c5bae359efc64d17b9591adc8dc7b185 (127.12.2.65:39893), f8641366ac034f14a249b162e30a5703 (127.12.2.67:41215)
I20260620 23:05:48.703996 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3145561ba344529e87b000f0a83e3e" candidate_uuid: "174c8d98e2bc45fcb171db3de048efbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5bae359efc64d17b9591adc8dc7b185"
I20260620 23:05:48.704061 19640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3145561ba344529e87b000f0a83e3e" candidate_uuid: "174c8d98e2bc45fcb171db3de048efbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8641366ac034f14a249b162e30a5703"
I20260620 23:05:48.704187 19349 raft_consensus.cc:3037] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:48.704242 19640 raft_consensus.cc:3037] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:48.707273 19640 raft_consensus.cc:2454] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174c8d98e2bc45fcb171db3de048efbf in term 1.
I20260620 23:05:48.707631 19349 raft_consensus.cc:2454] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174c8d98e2bc45fcb171db3de048efbf in term 1.
I20260620 23:05:48.707641 19426 leader_election.cc:302] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174c8d98e2bc45fcb171db3de048efbf, f8641366ac034f14a249b162e30a5703; no voters:
I20260620 23:05:48.707897 19725 raft_consensus.cc:2781] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:48.708313 19725 raft_consensus.cc:686] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 1 LEADER]: Becoming Leader. State: Replica: 174c8d98e2bc45fcb171db3de048efbf, State: Running, Role: LEADER
I20260620 23:05:48.708796 19725 consensus_queue.cc:237] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } } peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } }
I20260620 23:05:48.713287 18432 catalog_manager.cc:5495] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 174c8d98e2bc45fcb171db3de048efbf (127.12.2.66). New cstate: current_term: 1 leader_uuid: "174c8d98e2bc45fcb171db3de048efbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } health_report { overall_health: UNKNOWN } } }
W20260620 23:05:48.761312 19402 tablet.cc:2369] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:48.765038 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.759508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.12.2.66:46755)
0620 23:05:48.759697 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.759704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.759828 (+ 124us) server_negotiation.cc:407] Connection header received
0620 23:05:48.760038 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.760042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.760116 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.760261 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:48.760646 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.761315 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.762610 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.763454 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.763564 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.764011 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:48.764509 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:48.764634 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.764713 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.764789 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:05:48.772892 19349 raft_consensus.cc:1261] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185 [term 1 FOLLOWER]: Refusing update from remote peer 174c8d98e2bc45fcb171db3de048efbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:48.772894 19640 raft_consensus.cc:1261] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703 [term 1 FOLLOWER]: Refusing update from remote peer 174c8d98e2bc45fcb171db3de048efbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:48.773765 19730 consensus_queue.cc:1035] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:48.774109 19725 consensus_queue.cc:1035] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:48.786315 19735 mvcc.cc:205] Tried to move back new op lower bound from 7299058682964520960 to 7299058682714566656. Current Snapshot: MvccSnapshot[applied={T|T < 7299058682964520960}]
I20260620 23:05:48.806540 18432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:48.806869 18432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:48.820086 18432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07a99eb094dc4ae295ad268a70895388: Sending DeleteTablet for 3 replicas of tablet 4e3145561ba344529e87b000f0a83e3e
I20260620 23:05:48.821137 19470 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3145561ba344529e87b000f0a83e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:48 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:41878
I20260620 23:05:48.821427 19329 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3145561ba344529e87b000f0a83e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:48 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:40054
I20260620 23:05:48.823239 19620 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3145561ba344529e87b000f0a83e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:48 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:33308
I20260620 23:05:48.838310 19745 tablet_replica.cc:330] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: stopping tablet replica
I20260620 23:05:48.838893 19743 tablet_replica.cc:330] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf: stopping tablet replica
I20260620 23:05:48.839383 19743 raft_consensus.cc:2229] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:48.840531 19745 raft_consensus.cc:2229] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:48.840852 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 18400
I20260620 23:05:48.841046 19745 raft_consensus.cc:2258] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:48.842577 18462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:48.844086 19744 tablet_replica.cc:330] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: stopping tablet replica
I20260620 23:05:48.844467 19744 raft_consensus.cc:2229] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:48.844961 19744 raft_consensus.cc:2258] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:48.840076 19743 raft_consensus.cc:2258] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:48.850631 19744 ts_tablet_manager.cc:1905] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:48.851876 19743 ts_tablet_manager.cc:1905] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:48.857182 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.826023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.12.2.125:42515)
0620 23:05:48.826200 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.826206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.839886 (+ 13680us) server_negotiation.cc:407] Connection header received
0620 23:05:48.840163 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.840181 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.840263 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.840338 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:48.843346 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.844402 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.846059 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.846691 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.846811 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.850628 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:48.850634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:48.850638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:48.850690 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:48.854028 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.854838 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.854843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.854846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.855009 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.855414 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.855419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.855422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.855766 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:48.855986 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.856622 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.856844 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:05:48.857673 19745 ts_tablet_manager.cc:1905] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:48.857723 19744 ts_tablet_manager.cc:1918] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:48.857872 19744 log.cc:1196] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/4e3145561ba344529e87b000f0a83e3e
I20260620 23:05:48.863361 19743 ts_tablet_manager.cc:1918] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:48.863745 19743 log.cc:1196] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/4e3145561ba344529e87b000f0a83e3e
I20260620 23:05:48.864346 19743 ts_tablet_manager.cc:1939] T 4e3145561ba344529e87b000f0a83e3e P 174c8d98e2bc45fcb171db3de048efbf: Deleting consensus metadata
I20260620 23:05:48.868898 19745 ts_tablet_manager.cc:1918] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:48.869081 19745 log.cc:1196] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/4e3145561ba344529e87b000f0a83e3e
I20260620 23:05:48.869630 19745 ts_tablet_manager.cc:1939] T 4e3145561ba344529e87b000f0a83e3e P f8641366ac034f14a249b162e30a5703: Deleting consensus metadata
I20260620 23:05:48.874434 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.840075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.12.2.124:33779)
0620 23:05:48.840270 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.840275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.858318 (+ 18043us) server_negotiation.cc:407] Connection header received
0620 23:05:48.858552 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.858558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.858635 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.858710 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:48.860646 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.861741 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.863643 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.864323 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.864424 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.867872 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:48.867878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:48.867883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:48.867924 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:48.871187 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.871980 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.871986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.871990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.872144 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:48.872603 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:48.872609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:48.872612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:48.873027 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:48.873263 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.873859 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.874080 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:05:48.875216 19744 ts_tablet_manager.cc:1939] T 4e3145561ba344529e87b000f0a83e3e P c5bae359efc64d17b9591adc8dc7b185: Deleting consensus metadata
I20260620 23:05:49.006817 18419 catalog_manager.cc:4841] TS f8641366ac034f14a249b162e30a5703 (127.12.2.67:41215): tablet 4e3145561ba344529e87b000f0a83e3e (table test-table [id=3386ad840dea4a75bf937481baf0cb1b]) successfully deleted
I20260620 23:05:49.006829 18418 catalog_manager.cc:4841] TS c5bae359efc64d17b9591adc8dc7b185 (127.12.2.65:39893): tablet 4e3145561ba344529e87b000f0a83e3e (table test-table [id=3386ad840dea4a75bf937481baf0cb1b]) successfully deleted
I20260620 23:05:49.007356 18418 catalog_manager.cc:4841] TS 174c8d98e2bc45fcb171db3de048efbf (127.12.2.66:46755): tablet 4e3145561ba344529e87b000f0a83e3e (table test-table [id=3386ad840dea4a75bf937481baf0cb1b]) successfully deleted
I20260620 23:05:49.008342 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 18400
I20260620 23:05:49.047399 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:49.029684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.12.2.125:42515)
0620 23:05:49.029887 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:49.029893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:49.032536 (+ 2643us) server_negotiation.cc:407] Connection header received
0620 23:05:49.032777 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:49.032783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:49.032879 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:49.032972 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:49.035145 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:49.035897 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:49.037579 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:49.038189 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:49.038311 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:49.041276 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:49.041282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:49.041285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:49.041321 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:49.044680 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:49.045481 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:49.045486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:49.045490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:49.045629 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:49.046036 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:49.046042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:49.046045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:49.046315 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:49.046540 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:49.046867 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 23:05:49.047098 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:05:49.047745 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:49.030044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.12.2.124:33779)
0620 23:05:49.030218 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:49.030223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:49.032275 (+ 2052us) server_negotiation.cc:407] Connection header received
0620 23:05:49.033131 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:49.033136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:49.033212 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:49.033281 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:49.035213 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:49.036307 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:49.037917 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:49.038314 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:49.038424 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:49.041759 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:49.041765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:49.041770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:49.041810 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:49.044559 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:49.045424 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:49.045430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:49.045434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:49.045587 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:49.045997 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:49.046002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:49.046006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:49.046355 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:49.046570 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:49.047179 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 23:05:49.047394 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:05:49.790705 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:49.789963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.65:46517)
0620 23:05:49.790403 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:49.790529 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":231,"threads_started":1}
W20260620 23:05:49.791157 19395 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:41439 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
I20260620 23:05:49.793301 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:49.792607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.66:53809)
0620 23:05:49.793033 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:49.793140 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":226,"threads_started":1}
W20260620 23:05:49.793606 19536 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:41439 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
I20260620 23:05:49.799521 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:49.798873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.67:46239)
0620 23:05:49.799254 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:49.799357 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":207,"threads_started":1}
W20260620 23:05:49.799813 19686 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:41439 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
I20260620 23:05:50.316537 19758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 07a99eb094dc4ae295ad268a70895388)
I20260620 23:05:50.316960 19758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } }
I20260620 23:05:50.318711 19758 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07a99eb094dc4ae295ad268a70895388 (127.12.2.126:41439), 8904843a27ca47bb81ab9e2ddf21123b (127.12.2.125:42515)
I20260620 23:05:50.319870 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.318759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:33392)
0620 23:05:50.319361 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.319586 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"threads_started":1}
W20260620 23:05:50.320732 19204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
I20260620 23:05:50.328677 19761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 07a99eb094dc4ae295ad268a70895388)
I20260620 23:05:50.328976 19761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } }
I20260620 23:05:50.329456 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.328612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:33402)
0620 23:05:50.328939 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.329152 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 23:05:50.329674 19204 leader_election.cc:334] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 07a99eb094dc4ae295ad268a70895388 (127.12.2.126:41439): Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
I20260620 23:05:50.330552 19761 leader_election.cc:288] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07a99eb094dc4ae295ad268a70895388 (127.12.2.126:41439), 3f9a7aaaa9174ba6a11acf9635519a8d (127.12.2.124:33779)
I20260620 23:05:50.335870 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.318966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42515 (local address 127.0.0.1:44970)
0620 23:05:50.320051 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.320072 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:50.320234 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:50.328919 (+ 8685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:50.328925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:50.328974 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:50.329207 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.329228 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.330505 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.330512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.332832 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.332854 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.334881 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.334891 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.335150 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.335200 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 23:05:50.335282 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 23:05:50.335461 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"thread_start_us":8413,"threads_started":1}
I20260620 23:05:50.337167 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.319049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.12.2.125:42515)
0620 23:05:50.328477 (+ 9428us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:50.328483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:50.328511 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:05:50.328604 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:50.328610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:50.328724 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:50.328851 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:50.329430 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.330350 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.333091 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.334638 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.336293 (+ 1655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.336422 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:50.336482 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 23:05:50.336637 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9295,"thread_start_us":9104,"threads_started":1}
I20260620 23:05:50.337941 19040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8904843a27ca47bb81ab9e2ddf21123b" is_pre_election: true
I20260620 23:05:50.338387 19040 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f9a7aaaa9174ba6a11acf9635519a8d in term 1.
I20260620 23:05:50.339077 19202 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f9a7aaaa9174ba6a11acf9635519a8d, 8904843a27ca47bb81ab9e2ddf21123b; no voters: 07a99eb094dc4ae295ad268a70895388
I20260620 23:05:50.339561 19758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:05:50.339689 19758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 07a99eb094dc4ae295ad268a70895388)
I20260620 23:05:50.339792 19758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:05:50.339987 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.330644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:33414)
0620 23:05:50.339168 (+ 8524us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.339617 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8411,"thread_start_us":7830,"threads_started":1}
W20260620 23:05:50.340211 19017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
I20260620 23:05:50.341977 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.341569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:33424)
0620 23:05:50.341735 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.341826 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:05:50.342193 19017 leader_election.cc:334] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 07a99eb094dc4ae295ad268a70895388 (127.12.2.126:41439): Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
I20260620 23:05:50.342592 19758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } }
I20260620 23:05:50.343271 19758 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [CANDIDATE]: Term 2 election: Requested vote from peers 07a99eb094dc4ae295ad268a70895388 (127.12.2.126:41439), 8904843a27ca47bb81ab9e2ddf21123b (127.12.2.125:42515)
I20260620 23:05:50.343705 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.343342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:33440)
0620 23:05:50.343489 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.343574 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 23:05:50.343793 19040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8904843a27ca47bb81ab9e2ddf21123b"
I20260620 23:05:50.344007 19040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:05:50.345001 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.344535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:33442)
0620 23:05:50.344727 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.344822 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 23:05:50.345238 19204 leader_election.cc:334] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 07a99eb094dc4ae295ad268a70895388 (127.12.2.126:41439): Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
I20260620 23:05:50.346341 19040 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f9a7aaaa9174ba6a11acf9635519a8d in term 2.
I20260620 23:05:50.346676 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.331094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.12.2.124:33779)
0620 23:05:50.339144 (+ 8050us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:50.339150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:50.339835 (+ 685us) server_negotiation.cc:407] Connection header received
0620 23:05:50.340075 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:50.340081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:50.340146 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:50.340234 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:50.340899 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.342137 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.344464 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.345666 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.345791 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.345903 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:50.346198 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:05:50.346329 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7891,"thread_start_us":7848,"threads_started":1}
I20260620 23:05:50.346689 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.330776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:33779 (local address 127.0.0.1:49060)
0620 23:05:50.339728 (+ 8952us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.339747 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:50.339951 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:50.340264 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:50.340269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:50.340308 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:50.340658 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.340673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.342324 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.342329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.344258 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.344270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.345845 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.345852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.346032 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.346058 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:05:50.346145 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:05:50.346284 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8838,"thread_start_us":8526,"threads_started":1}
I20260620 23:05:50.346753 19202 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f9a7aaaa9174ba6a11acf9635519a8d, 8904843a27ca47bb81ab9e2ddf21123b; no voters: 07a99eb094dc4ae295ad268a70895388
I20260620 23:05:50.347082 19758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:05:50.347462 19227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8904843a27ca47bb81ab9e2ddf21123b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" is_pre_election: true
I20260620 23:05:50.347585 19758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [term 2 LEADER]: Becoming Leader. State: Replica: 3f9a7aaaa9174ba6a11acf9635519a8d, State: Running, Role: LEADER
I20260620 23:05:50.347999 19758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } }
I20260620 23:05:50.348263 19014 leader_election.cc:302] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8904843a27ca47bb81ab9e2ddf21123b; no voters: 07a99eb094dc4ae295ad268a70895388, 3f9a7aaaa9174ba6a11acf9635519a8d
I20260620 23:05:50.348590 19761 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:05:50.350235 19768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9a7aaaa9174ba6a11acf9635519a8d. Latest consensus state: current_term: 2 leader_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:50.350608 19768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:50.351289 19770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:50.354838 19770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:50.355715 19770 catalog_manager.cc:1259] Loaded cluster ID: 1befd225d77342798bc52be4cecf56e2
I20260620 23:05:50.355803 19770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:50.357045 19770 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:50.357908 19770 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: Loaded TSK: 0
I20260620 23:05:50.358755 19770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:50.373258 19217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:50.380167 19217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:50.384148 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.383735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:33468)
0620 23:05:50.383894 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.383990 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:05:50.384233 19040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [term 2 FOLLOWER]: Refusing update from remote peer 3f9a7aaaa9174ba6a11acf9635519a8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 23:05:50.385046 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.384718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:33478)
0620 23:05:50.384842 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.384929 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 23:05:50.385006 19768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 23:05:50.385231 19204 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d -> Peer 07a99eb094dc4ae295ad268a70895388 (127.12.2.126:41439): Couldn't send request to peer 07a99eb094dc4ae295ad268a70895388. Status: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:05:50.387919 19761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9a7aaaa9174ba6a11acf9635519a8d. Latest consensus state: current_term: 2 leader_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:50.388206 19761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:05:50.388916 19768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:50.389187 19768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:50.389588 19761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "07a99eb094dc4ae295ad268a70895388" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41439 } } peers { permanent_uuid: "8904843a27ca47bb81ab9e2ddf21123b" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42515 } attrs { promote: false } } peers { permanent_uuid: "3f9a7aaaa9174ba6a11acf9635519a8d" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 33779 } attrs { promote: false } } }
I20260620 23:05:50.389842 19761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8904843a27ca47bb81ab9e2ddf21123b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:05:50.404189 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.397100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:46755 (local address 127.0.0.1:41896)
0620 23:05:50.397386 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.397412 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:50.397489 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:50.398218 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:50.398222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:50.398247 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:50.398591 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.398604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.399917 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.399922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.401816 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.401828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.403405 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.403411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.403615 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.403654 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:05:50.403720 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 23:05:50.403863 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":4}
I20260620 23:05:50.404984 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.397457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.12.2.66:46755)
0620 23:05:50.397916 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:50.397922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:50.397939 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:05:50.398005 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:50.398009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:50.398076 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:50.398745 (+ 669us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:50.398907 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.399759 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.402013 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.403170 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.404422 (+ 1252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.404525 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:50.404574 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 23:05:50.404702 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":230,"threads_started":1}
I20260620 23:05:50.405112 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.397724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39893 (local address 127.0.0.1:40068)
0620 23:05:50.398274 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.398293 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:50.398424 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:50.399594 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:50.399598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:50.399625 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:50.399962 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.399975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.401341 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.401346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.402861 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.402872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.404532 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.404538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.404650 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.404673 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:05:50.404730 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 23:05:50.404813 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":1,"spinlock_wait_cycles":12160,"thread_start_us":560,"threads_started":1}
I20260620 23:05:50.405755 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.398184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.12.2.65:39893)
0620 23:05:50.399304 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:50.399309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:50.399328 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:05:50.399393 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:50.399397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:50.399467 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:50.399565 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:50.400163 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.401194 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.403267 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.404346 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.405331 (+ 985us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.405426 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:50.405471 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 23:05:50.405560 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":894,"threads_started":1}
I20260620 23:05:50.405643 19470 tablet_service.cc:1467] Processing CreateTablet for tablet 805f5e8cbbcd4b35adce10a0313a9097 (DEFAULT_TABLE table=test-table [id=19c45dc2820c4c8d9c67c3e33c0e4a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:50.405856 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.396679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:41215 (local address 127.0.0.1:33310)
0620 23:05:50.396973 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.396992 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:50.397094 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:50.400304 (+ 3210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:50.400309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:50.400332 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:50.400543 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.400552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.401720 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.401725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.403669 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.403680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.405251 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.405256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.405419 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.405449 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:05:50.405509 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:05:50.405617 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":57}
I20260620 23:05:50.406112 19470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805f5e8cbbcd4b35adce10a0313a9097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:50.406579 19329 tablet_service.cc:1467] Processing CreateTablet for tablet 805f5e8cbbcd4b35adce10a0313a9097 (DEFAULT_TABLE table=test-table [id=19c45dc2820c4c8d9c67c3e33c0e4a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:50.406955 19329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805f5e8cbbcd4b35adce10a0313a9097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:50.406971 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.396812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.12.2.67:41215)
0620 23:05:50.399975 (+ 3163us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:50.399981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:50.399999 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:05:50.400062 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:50.400066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:50.400146 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:50.400651 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:50.400786 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.401578 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.403826 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.405073 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.406411 (+ 1338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.406508 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:50.406554 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 23:05:50.406676 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3059,"thread_start_us":2973,"threads_started":1}
I20260620 23:05:50.407548 19620 tablet_service.cc:1467] Processing CreateTablet for tablet 805f5e8cbbcd4b35adce10a0313a9097 (DEFAULT_TABLE table=test-table [id=19c45dc2820c4c8d9c67c3e33c0e4a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:50.407931 19620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 805f5e8cbbcd4b35adce10a0313a9097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:50.418221 19781 tablet_bootstrap.cc:492] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf: Bootstrap starting.
I20260620 23:05:50.418224 19782 tablet_bootstrap.cc:492] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185: Bootstrap starting.
I20260620 23:05:50.418593 19783 tablet_bootstrap.cc:492] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703: Bootstrap starting.
I20260620 23:05:50.419847 19782 tablet_bootstrap.cc:654] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:50.420202 19783 tablet_bootstrap.cc:654] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:50.420753 19781 tablet_bootstrap.cc:654] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:50.422183 19783 tablet_bootstrap.cc:492] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703: No bootstrap required, opened a new log
I20260620 23:05:50.422254 19782 tablet_bootstrap.cc:492] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185: No bootstrap required, opened a new log
I20260620 23:05:50.422330 19783 ts_tablet_manager.cc:1397] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:05:50.422391 19782 ts_tablet_manager.cc:1397] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:05:50.422756 19781 tablet_bootstrap.cc:492] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf: No bootstrap required, opened a new log
I20260620 23:05:50.422888 19781 ts_tablet_manager.cc:1397] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 23:05:50.423076 19782 raft_consensus.cc:348] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } } peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } }
I20260620 23:05:50.423082 19783 raft_consensus.cc:348] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } } peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } }
I20260620 23:05:50.423257 19782 raft_consensus.cc:374] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:50.423316 19783 raft_consensus.cc:374] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:50.423337 19782 raft_consensus.cc:729] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5bae359efc64d17b9591adc8dc7b185, State: Initialized, Role: FOLLOWER
I20260620 23:05:50.423409 19783 raft_consensus.cc:729] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8641366ac034f14a249b162e30a5703, State: Initialized, Role: FOLLOWER
I20260620 23:05:50.423564 19782 consensus_queue.cc:260] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } } peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } }
I20260620 23:05:50.423657 19783 consensus_queue.cc:260] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } } peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } }
I20260620 23:05:50.423643 19781 raft_consensus.cc:348] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } } peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } }
I20260620 23:05:50.423926 19781 raft_consensus.cc:374] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:50.424029 19781 raft_consensus.cc:729] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174c8d98e2bc45fcb171db3de048efbf, State: Initialized, Role: FOLLOWER
I20260620 23:05:50.424283 19781 consensus_queue.cc:260] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } } peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } }
I20260620 23:05:50.425619 19782 ts_tablet_manager.cc:1428] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:50.427852 19394 heartbeater.cc:503] Master 127.12.2.124:33779 was elected leader, sending a full tablet report...
I20260620 23:05:50.429605 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.426202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.65:36043)
0620 23:05:50.428316 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.428492 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1947,"thread_start_us":1804,"threads_started":1}
I20260620 23:05:50.431201 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.430334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.65:54599)
0620 23:05:50.430786 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.430964 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"threads_started":1}
I20260620 23:05:50.432793 19781 ts_tablet_manager.cc:1428] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf: Time spent starting tablet: real 0.010s user 0.000s sys 0.005s
I20260620 23:05:50.433037 19535 heartbeater.cc:503] Master 127.12.2.124:33779 was elected leader, sending a full tablet report...
I20260620 23:05:50.435328 19783 ts_tablet_manager.cc:1428] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20260620 23:05:50.436077 19685 heartbeater.cc:503] Master 127.12.2.124:33779 was elected leader, sending a full tablet report...
I20260620 23:05:50.436331 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.430802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.66:52673)
0620 23:05:50.435923 (+ 5121us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.436069 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4988,"thread_start_us":4851,"threads_started":1}
I20260620 23:05:50.437225 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.436854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.66:38187)
0620 23:05:50.437007 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.437091 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:05:50.443462 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.434908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.67:47929)
0620 23:05:50.443094 (+ 8186us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.443247 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8047,"thread_start_us":7880,"threads_started":1}
I20260620 23:05:50.444388 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.444073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.67:47101)
0620 23:05:50.444209 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.444287 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 23:05:50.471235 19788 raft_consensus.cc:482] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:50.471429 19788 raft_consensus.cc:504] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } } peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } }
I20260620 23:05:50.472937 19788 leader_election.cc:288] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 174c8d98e2bc45fcb171db3de048efbf (127.12.2.66:46755), c5bae359efc64d17b9591adc8dc7b185 (127.12.2.65:39893)
I20260620 23:05:50.478832 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.473097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:46755 (local address 127.12.2.67:50065)
0620 23:05:50.473236 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.473254 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:50.473393 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:50.473859 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:50.473875 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:50.473904 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:50.474140 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.474152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.475428 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.475433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.476975 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.476986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.478117 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.478123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.478299 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.478339 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:05:50.478433 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:05:50.478531 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":47}
I20260620 23:05:50.478878 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.473227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:50065 (local address 127.12.2.66:46755)
0620 23:05:50.473421 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:50.473427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:50.473476 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:05:50.473549 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:50.473555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:50.473643 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:50.473737 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:50.474374 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.475281 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.477119 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.477953 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.478044 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.478121 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:50.478465 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 23:05:50.478589 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:05:50.478981 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.473567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:34585 (local address 127.12.2.65:39893)
0620 23:05:50.473760 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:50.473765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:50.474052 (+ 287us) server_negotiation.cc:407] Connection header received
0620 23:05:50.474160 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:50.474165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:50.474236 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:50.474313 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:50.474843 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.475646 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.477358 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.478150 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.478266 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.478365 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:50.478631 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:05:50.478746 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:05:50.479148 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.473436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39893 (local address 127.12.2.67:34585)
0620 23:05:50.473950 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.473966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:50.474063 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:50.474373 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:50.474378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:50.474403 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:50.474697 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.474709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.475756 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.475760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.477224 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:50.477236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.478312 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.478316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:50.478484 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.478514 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:05:50.478594 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:05:50.478706 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":18,"thread_start_us":283,"threads_started":1}
I20260620 23:05:50.479737 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805f5e8cbbcd4b35adce10a0313a9097" candidate_uuid: "f8641366ac034f14a249b162e30a5703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5bae359efc64d17b9591adc8dc7b185" is_pre_election: true
I20260620 23:05:50.479653 19490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805f5e8cbbcd4b35adce10a0313a9097" candidate_uuid: "f8641366ac034f14a249b162e30a5703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174c8d98e2bc45fcb171db3de048efbf" is_pre_election: true
I20260620 23:05:50.479962 19349 raft_consensus.cc:2454] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8641366ac034f14a249b162e30a5703 in term 0.
I20260620 23:05:50.480046 19490 raft_consensus.cc:2454] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8641366ac034f14a249b162e30a5703 in term 0.
I20260620 23:05:50.480528 19575 leader_election.cc:302] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174c8d98e2bc45fcb171db3de048efbf, f8641366ac034f14a249b162e30a5703; no voters:
I20260620 23:05:50.480924 19788 raft_consensus.cc:2781] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:50.481045 19788 raft_consensus.cc:482] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:50.481141 19788 raft_consensus.cc:3037] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:50.483244 19788 raft_consensus.cc:504] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } } peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } }
I20260620 23:05:50.483772 19788 leader_election.cc:288] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [CANDIDATE]: Term 1 election: Requested vote from peers 174c8d98e2bc45fcb171db3de048efbf (127.12.2.66:46755), c5bae359efc64d17b9591adc8dc7b185 (127.12.2.65:39893)
I20260620 23:05:50.484215 19490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805f5e8cbbcd4b35adce10a0313a9097" candidate_uuid: "f8641366ac034f14a249b162e30a5703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174c8d98e2bc45fcb171db3de048efbf"
I20260620 23:05:50.484308 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "805f5e8cbbcd4b35adce10a0313a9097" candidate_uuid: "f8641366ac034f14a249b162e30a5703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5bae359efc64d17b9591adc8dc7b185"
I20260620 23:05:50.484376 19490 raft_consensus.cc:3037] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:50.484506 19349 raft_consensus.cc:3037] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:50.486486 19349 raft_consensus.cc:2454] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8641366ac034f14a249b162e30a5703 in term 1.
I20260620 23:05:50.486614 19490 raft_consensus.cc:2454] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8641366ac034f14a249b162e30a5703 in term 1.
I20260620 23:05:50.486872 19575 leader_election.cc:302] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5bae359efc64d17b9591adc8dc7b185, f8641366ac034f14a249b162e30a5703; no voters:
I20260620 23:05:50.487141 19788 raft_consensus.cc:2781] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:50.487524 19788 raft_consensus.cc:686] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 1 LEADER]: Becoming Leader. State: Replica: f8641366ac034f14a249b162e30a5703, State: Running, Role: LEADER
I20260620 23:05:50.487890 19788 consensus_queue.cc:237] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } } peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } }
I20260620 23:05:50.488441 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.488051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.67:54717)
0620 23:05:50.488208 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.488315 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:05:50.491101 19217 catalog_manager.cc:5495] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8641366ac034f14a249b162e30a5703 (127.12.2.67). New cstate: current_term: 1 leader_uuid: "f8641366ac034f14a249b162e30a5703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8641366ac034f14a249b162e30a5703" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 41215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:50.510506 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.505829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.12.2.67:41215)
0620 23:05:50.506021 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:50.506026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:50.506042 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:05:50.506241 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:50.506245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:50.506343 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:50.506483 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:50.506745 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.507512 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.508732 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.509123 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.509217 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.509528 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:50.510042 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:50.510142 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:50.510180 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 23:05:50.510250 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:05:50.514146 19349 raft_consensus.cc:1261] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185 [term 1 FOLLOWER]: Refusing update from remote peer f8641366ac034f14a249b162e30a5703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:50.514526 19490 raft_consensus.cc:1261] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf [term 1 FOLLOWER]: Refusing update from remote peer f8641366ac034f14a249b162e30a5703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:50.514799 19795 consensus_queue.cc:1035] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5bae359efc64d17b9591adc8dc7b185" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:50.515208 19788 consensus_queue.cc:1035] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "174c8d98e2bc45fcb171db3de048efbf" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:50.517449 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.516906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.65:46873)
0620 23:05:50.517149 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.517243 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 23:05:50.518040 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.517150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.66:51145)
0620 23:05:50.517721 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.517850 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440}
I20260620 23:05:50.523046 19800 mvcc.cc:205] Tried to move back new op lower bound from 7299058690099785728 to 7299058690002116608. Current Snapshot: MvccSnapshot[applied={T|T < 7299058690099785728}]
I20260620 23:05:50.524540 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.524051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.66:34821)
0620 23:05:50.524304 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.524397 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260620 23:05:50.524887 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.524298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.67:52317)
0620 23:05:50.524556 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.524668 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260620 23:05:50.530725 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.529814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.67:33805)
0620 23:05:50.530424 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.530537 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493}
I20260620 23:05:50.531883 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.531399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.12.2.65:48033)
0620 23:05:50.531619 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.531716 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 23:05:50.539489 19217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:50.539856 19217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:50.550338 19217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f9a7aaaa9174ba6a11acf9635519a8d: Sending DeleteTablet for 3 replicas of tablet 805f5e8cbbcd4b35adce10a0313a9097
I20260620 23:05:50.550525 19329 tablet_service.cc:1514] Processing DeleteTablet for tablet 805f5e8cbbcd4b35adce10a0313a9097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:50 UTC) from {username='oryx', principal='oryx/127.12.2.124@KRBTEST.COM'} at 127.0.0.1:40068
I20260620 23:05:50.551450 19808 tablet_replica.cc:330] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185: stopping tablet replica
I20260620 23:05:50.551483 19620 tablet_service.cc:1514] Processing DeleteTablet for tablet 805f5e8cbbcd4b35adce10a0313a9097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:50 UTC) from {username='oryx', principal='oryx/127.12.2.124@KRBTEST.COM'} at 127.0.0.1:33310
I20260620 23:05:50.551614 19808 raft_consensus.cc:2229] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:50.551903 19808 raft_consensus.cc:2258] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:50.551913 19470 tablet_service.cc:1514] Processing DeleteTablet for tablet 805f5e8cbbcd4b35adce10a0313a9097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:50 UTC) from {username='oryx', principal='oryx/127.12.2.124@KRBTEST.COM'} at 127.0.0.1:41896
I20260620 23:05:50.552417 19809 tablet_replica.cc:330] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703: stopping tablet replica
I20260620 23:05:50.552634 19809 raft_consensus.cc:2229] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:50.552799 19808 ts_tablet_manager.cc:1905] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:50.553299 19809 raft_consensus.cc:2258] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:50.554278 19809 ts_tablet_manager.cc:1905] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:50.556957 19329 tablet_service.cc:1514] Processing DeleteTablet for tablet 805f5e8cbbcd4b35adce10a0313a9097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:50 UTC) from {username='oryx', principal='oryx/127.12.2.124@KRBTEST.COM'} at 127.0.0.1:40068
I20260620 23:05:50.559127 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 19189
I20260620 23:05:50.559448 19808 ts_tablet_manager.cc:1918] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:50.559577 19808 log.cc:1196] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/805f5e8cbbcd4b35adce10a0313a9097
I20260620 23:05:50.559991 19808 ts_tablet_manager.cc:1939] T 805f5e8cbbcd4b35adce10a0313a9097 P c5bae359efc64d17b9591adc8dc7b185: Deleting consensus metadata
I20260620 23:05:50.561401 19809 ts_tablet_manager.cc:1918] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:50.561609 19809 log.cc:1196] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/805f5e8cbbcd4b35adce10a0313a9097
I20260620 23:05:50.562103 19809 ts_tablet_manager.cc:1939] T 805f5e8cbbcd4b35adce10a0313a9097 P f8641366ac034f14a249b162e30a5703: Deleting consensus metadata
W20260620 23:05:50.562263 19203 catalog_manager.cc:4808] TS c5bae359efc64d17b9591adc8dc7b185 (127.12.2.65:39893): delete failed for tablet 805f5e8cbbcd4b35adce10a0313a9097 because the tablet was not found. No further retry: Not found: Tablet not found: 805f5e8cbbcd4b35adce10a0313a9097
I20260620 23:05:50.563225 19204 catalog_manager.cc:4841] TS f8641366ac034f14a249b162e30a5703 (127.12.2.67:41215): tablet 805f5e8cbbcd4b35adce10a0313a9097 (table test-table [id=19c45dc2820c4c8d9c67c3e33c0e4a2c]) successfully deleted
I20260620 23:05:50.567364 19810 tablet_replica.cc:330] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf: stopping tablet replica
I20260620 23:05:50.567572 19810 raft_consensus.cc:2229] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:50.567871 19810 raft_consensus.cc:2258] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:50.568789 19810 ts_tablet_manager.cc:1905] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:50.574121 19810 ts_tablet_manager.cc:1918] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:50.574260 19810 log.cc:1196] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/805f5e8cbbcd4b35adce10a0313a9097
I20260620 23:05:50.574643 19810 ts_tablet_manager.cc:1939] T 805f5e8cbbcd4b35adce10a0313a9097 P 174c8d98e2bc45fcb171db3de048efbf: Deleting consensus metadata
I20260620 23:05:50.575630 19203 catalog_manager.cc:4841] TS 174c8d98e2bc45fcb171db3de048efbf (127.12.2.66:46755): tablet 805f5e8cbbcd4b35adce10a0313a9097 (table test-table [id=19c45dc2820c4c8d9c67c3e33c0e4a2c]) successfully deleted
I20260620 23:05:50.582187 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.560661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.12.2.65:39893)
0620 23:05:50.560836 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:50.560842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:50.567343 (+ 6501us) server_negotiation.cc:407] Connection header received
0620 23:05:50.567547 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:50.567552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:50.567635 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:50.567744 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:50.569609 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.570657 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.572448 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.573066 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.573192 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.576463 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:50.576479 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:50.576495 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:50.576535 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:50.579375 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:50.580192 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:50.580199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:50.580202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:50.580317 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:50.580678 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:50.580681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:50.580683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:50.581052 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:50.581320 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:50.581636 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 23:05:50.581832 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:05:50.583146 19389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:50.674997 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 19189
I20260620 23:05:50.696887 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 19407
I20260620 23:05:50.709677 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.697590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.12.2.66:46755)
0620 23:05:50.697784 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:50.697790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:50.697871 (+ 81us) server_negotiation.cc:407] Connection header received
0620 23:05:50.698067 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:50.698073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:50.698142 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:50.698236 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:50.699505 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.700191 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.701403 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.701774 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.701880 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.704303 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:50.704316 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:50.704332 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:50.704364 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:50.707118 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:50.707742 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:50.707752 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:50.707754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:50.707873 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:50.708214 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:50.708219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:50.708222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:50.708592 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:50.708860 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:50.709155 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:05:50.709358 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:05:50.710253 19530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:50.826726 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 19407
I20260620 23:05:50.844648 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 19557
I20260620 23:05:50.865906 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.845368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.12.2.67:41215)
0620 23:05:50.845557 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:50.845562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:50.845644 (+ 82us) server_negotiation.cc:407] Connection header received
0620 23:05:50.845807 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:50.845812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:50.845926 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:50.846036 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:50.847729 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.849001 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.850779 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:50.851507 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:50.851640 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:50.858699 (+ 7059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:50.858717 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:50.858739 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:50.858817 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:50.863082 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:50.863763 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:50.863772 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:50.863776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:50.863955 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:50.864321 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:50.864327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:50.864330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:50.864793 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:50.865095 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:50.865292 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 23:05:50.865509 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:05:50.866617 19680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:50.966089 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.965318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:33482)
0620 23:05:50.965818 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.965921 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
I20260620 23:05:50.967422 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:50.967074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:41439 (local address 127.0.0.1:33486)
0620 23:05:50.967222 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:50.967294 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:41439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 23:05:50.993441 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 19557
I20260620 23:05:51.010392 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 18893
I20260620 23:05:51.011430 19060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:51.124316 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 18893
I20260620 23:05:51.139770 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 18993
I20260620 23:05:51.140705 19247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:51.256809 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 18993
2026-06-20T23:05:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9030 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:51 dist-test-slave-7g8g krb5kdc[19821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:51 dist-test-slave-7g8g krb5kdc[19821](info): set up 2 sockets
Jun 20 23:05:51 dist-test-slave-7g8g krb5kdc[19821](info): commencing operation
krb5kdc: starting...
W20260620 23:05:52.263813 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:52 dist-test-slave-7g8g krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.126.keytab.
Entry for principal oryx/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.126.keytab.
I20260620 23:05:52.459381 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:37269
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.126.keytab
--principal=oryx/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:37269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:52.765544 19834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:52.765863 19834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:52.766000 19834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:52.773170 19834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:52.773289 19834 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:05:52.773350 19834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:52.773434 19834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:52.773500 19834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:52.773581 19834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:52.783433 19834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:37269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.126.keytab
--principal=oryx/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:37269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:52.784723 19834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:52.786031 19834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:52.797181 19839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:52.797214 19840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:52.797367 19842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:52.798398 19834 server_base.cc:1034] running on GCE node
I20260620 23:05:52.799192 19834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:52.800563 19834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:52.801780 19834 hybrid_clock.cc:648] HybridClock initialized: now 1781996752801706 us; error 40 us; skew 500 ppm
I20260620 23:05:52.802091 19834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:52 dist-test-slave-7g8g krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996752, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:52.805053 19834 init.cc:377] Logged in from keytab as oryx/127.12.2.126@KRBTEST.COM (short username oryx)
I20260620 23:05:52.806442 19834 webserver.cc:460] Webserver started at http://127.12.2.126:37767/ using document root <none> and password file <none>
I20260620 23:05:52.807157 19834 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:52.807255 19834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:52.807548 19834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:52.810348 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "1a9df4e2ca774683b1f0014599bcc7e5"
format_stamp: "Formatted at 2026-06-20 23:05:52 on dist-test-slave-7g8g"
server_key: "f09151e27562ea13472ab29c401f93ee"
server_key_iv: "a4fd47daa538aa95e1c82370d89e0587"
server_key_version: "encryptionkey@0"
I20260620 23:05:52.811141 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "1a9df4e2ca774683b1f0014599bcc7e5"
format_stamp: "Formatted at 2026-06-20 23:05:52 on dist-test-slave-7g8g"
server_key: "f09151e27562ea13472ab29c401f93ee"
server_key_iv: "a4fd47daa538aa95e1c82370d89e0587"
server_key_version: "encryptionkey@0"
I20260620 23:05:52.816038 19834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:05:52.819344 19849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:52.820617 19834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:05:52.820773 19834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "1a9df4e2ca774683b1f0014599bcc7e5"
format_stamp: "Formatted at 2026-06-20 23:05:52 on dist-test-slave-7g8g"
server_key: "f09151e27562ea13472ab29c401f93ee"
server_key_iv: "a4fd47daa538aa95e1c82370d89e0587"
server_key_version: "encryptionkey@0"
I20260620 23:05:52.820955 19834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:52.838143 19834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:52.842140 19834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:52.842371 19834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:52.862059 19834 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:37269
I20260620 23:05:52.862082 19910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:37269 every 8 connection(s)
I20260620 23:05:52.863428 19834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:52.867203 19911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:52.868069 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 19834
I20260620 23:05:52.868274 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:52.869051 12297 external_mini_cluster.cc:1428] Setting key dabb7bc85f48c0396d0098b66a35b9c4
I20260620 23:05:52.881006 19911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5: Bootstrap starting.
Jun 20 23:05:52 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:52.884684 19911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:52.885797 19911 log.cc:824] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:52.888895 19911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5: No bootstrap required, opened a new log
I20260620 23:05:52.891307 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:52.871095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.12.2.126:37269)
0620 23:05:52.871882 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:52.871893 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:52.871937 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:05:52.873568 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:52.873593 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:52.874008 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:52.874351 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:52.876066 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:52.877339 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:52.879091 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:52.879737 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:52.879868 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:52.884325 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:52.884333 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:52.884348 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:52.884383 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:52.887483 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:52.888208 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:52.888213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:52.888219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:52.888389 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:52.888753 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:52.888756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:52.888758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:52.889287 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:52.889493 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:52.889907 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 23:05:52.890226 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":362,"threads_started":1}
I20260620 23:05:52.897153 19911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9df4e2ca774683b1f0014599bcc7e5" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37269 } }
I20260620 23:05:52.897497 19911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:52.897569 19911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9df4e2ca774683b1f0014599bcc7e5, State: Initialized, Role: FOLLOWER
I20260620 23:05:52.898028 19911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9df4e2ca774683b1f0014599bcc7e5" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37269 } }
I20260620 23:05:52.898216 19911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:52.898314 19911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:52.898432 19911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:52.900480 19911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9df4e2ca774683b1f0014599bcc7e5" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37269 } }
I20260620 23:05:52.900902 19911 leader_election.cc:302] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a9df4e2ca774683b1f0014599bcc7e5; no voters:
I20260620 23:05:52.901371 19911 leader_election.cc:288] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:52.901535 19916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:52.902514 19916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9df4e2ca774683b1f0014599bcc7e5, State: Running, Role: LEADER
I20260620 23:05:52.903065 19916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9df4e2ca774683b1f0014599bcc7e5" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37269 } }
I20260620 23:05:52.903375 19911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:52.905802 19918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a9df4e2ca774683b1f0014599bcc7e5. Latest consensus state: current_term: 1 leader_uuid: "1a9df4e2ca774683b1f0014599bcc7e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9df4e2ca774683b1f0014599bcc7e5" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37269 } } }
I20260620 23:05:52.905798 19917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a9df4e2ca774683b1f0014599bcc7e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9df4e2ca774683b1f0014599bcc7e5" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37269 } } }
I20260620 23:05:52.906219 19917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:52.906210 19918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:52.907917 19925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:52.912958 19925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:52.920740 19925 catalog_manager.cc:1347] Generated new cluster ID: 1d5e9e809b0e43349b7e8cfaa3c6bdb5
I20260620 23:05:52.920881 19925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:52.927263 19925 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:52.928561 19925 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:52.939273 19925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5: Generated new TSK 0
I20260620 23:05:52.940081 19925 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab.
Entry for principal oryx/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab.
I20260620 23:05:52.993064 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37269
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab
--principal=oryx/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:53.318301 19937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:53.318615 19937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:53.318765 19937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:53.326195 19937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:53.326321 19937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:53.326436 19937 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:05:53.326545 19937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:05:53.336608 19937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab
--principal=oryx/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.2.126:37269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:53.337917 19937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:53.339274 19937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:53.352274 19942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:53.352447 19945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:53.352298 19943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:53.353037 19937 server_base.cc:1034] running on GCE node
I20260620 23:05:53.353690 19937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:53.354674 19937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:53.355908 19937 hybrid_clock.cc:648] HybridClock initialized: now 1781996753355829 us; error 56 us; skew 500 ppm
I20260620 23:05:53.356185 19937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:53 dist-test-slave-7g8g krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996753, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:53.359263 19937 init.cc:377] Logged in from keytab as oryx/127.12.2.65@KRBTEST.COM (short username oryx)
I20260620 23:05:53.360615 19937 webserver.cc:460] Webserver started at http://127.12.2.65:39443/ using document root <none> and password file <none>
I20260620 23:05:53.361250 19937 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:53.361331 19937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:53.361580 19937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:53.364468 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "1dd320521b0a45949654b6ad5163b6cb"
format_stamp: "Formatted at 2026-06-20 23:05:53 on dist-test-slave-7g8g"
server_key: "42f6a913292aece4ae4605b81088c7ba"
server_key_iv: "4d94f18330a459e44845993197f3a94a"
server_key_version: "encryptionkey@0"
I20260620 23:05:53.365196 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "1dd320521b0a45949654b6ad5163b6cb"
format_stamp: "Formatted at 2026-06-20 23:05:53 on dist-test-slave-7g8g"
server_key: "42f6a913292aece4ae4605b81088c7ba"
server_key_iv: "4d94f18330a459e44845993197f3a94a"
server_key_version: "encryptionkey@0"
I20260620 23:05:53.370117 19937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:05:53.373374 19952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:53.374677 19937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:53.374802 19937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "1dd320521b0a45949654b6ad5163b6cb"
format_stamp: "Formatted at 2026-06-20 23:05:53 on dist-test-slave-7g8g"
server_key: "42f6a913292aece4ae4605b81088c7ba"
server_key_iv: "4d94f18330a459e44845993197f3a94a"
server_key_version: "encryptionkey@0"
I20260620 23:05:53.374997 19937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:53.404721 19937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:53.408703 19937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:53.408941 19937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:53.411382 19937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:53.411477 19937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:53.411597 19937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:53.411664 19937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:05:53 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996753, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:53.432061 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:53.412024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:42931 (local address 127.12.2.126:37269)
0620 23:05:53.412491 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:53.412495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:53.413793 (+ 1298us) server_negotiation.cc:407] Connection header received
0620 23:05:53.415782 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:53.415787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:53.415866 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:53.415957 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:53.417829 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.418778 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.421107 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.421616 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.421760 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:53.426272 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:53.426279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:53.426283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:53.426325 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:53.428869 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:53.429801 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:53.429805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:53.429807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:53.429916 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:53.430370 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:53.430373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:53.430375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:53.430632 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:53.430745 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:53.431655 (+ 910us) server_negotiation.cc:299] Negotiation successful
0620 23:05:53.431819 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":268,"threads_started":1}
I20260620 23:05:53.433693 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:53.412666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37269 (local address 127.12.2.65:42931)
0620 23:05:53.413465 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:53.413500 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:53.415511 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:53.416189 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:53.416197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:53.416836 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:53.417652 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:53.417680 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.418971 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.418976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:53.420974 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:53.420986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.421873 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.421880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:53.422096 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:53.422770 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:53.422800 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:53.426006 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:53.429013 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:53.429022 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:53.429042 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:53.429662 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:53.430008 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:53.430013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:53.430017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:53.430259 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:53.430743 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:53.430752 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:53.431241 (+ 489us) client_negotiation.cc:769] Sending connection context
0620 23:05:53.431566 (+ 325us) client_negotiation.cc:240] Negotiation successful
0620 23:05:53.432161 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":348,"threads_started":1}
I20260620 23:05:53.453698 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:53.440260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37269 (local address 127.12.2.65:43709)
0620 23:05:53.440948 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:53.440970 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:53.441121 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:53.441449 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:53.441454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:53.442040 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:53.442405 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:53.442418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.443426 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.443431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:53.445028 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:53.445042 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.445815 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.445821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:53.446018 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:53.446755 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:53.446780 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:53.447754 (+ 974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:53.451110 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:53.451116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:53.451120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:53.451718 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:53.452072 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:53.452076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:53.452079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:53.452246 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:53.452857 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:53.452860 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:53.453005 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 23:05:53.453119 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:05:53.453373 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":585,"threads_started":1}
I20260620 23:05:53.453791 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:53.440679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:43709 (local address 127.12.2.126:37269)
0620 23:05:53.440842 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:53.440846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:53.441207 (+ 361us) server_negotiation.cc:407] Connection header received
0620 23:05:53.441267 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:53.441271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:53.441331 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:53.441406 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:53.442557 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.443296 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.445175 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.445640 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.445736 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:53.447991 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:53.447998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:53.448003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:53.448043 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:53.450991 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:53.451830 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:53.451835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:53.451839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:53.451972 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:53.452392 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:53.452397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:53.452401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:53.452764 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:53.452900 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:53.453237 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 23:05:53.453460 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:05:53.460529 19937 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:43785
I20260620 23:05:53.460552 20072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:43785 every 8 connection(s)
I20260620 23:05:53.461661 19937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:53.463404 20073 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37269
I20260620 23:05:53.463603 20073 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:53.464172 20073 heartbeater.cc:511] Master 127.12.2.126:37269 requested a full tablet report, sending...
I20260620 23:05:53.465879 19866 ts_manager.cc:194] Registered new tserver with Master: 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65:43785)
I20260620 23:05:53.465962 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 19937
I20260620 23:05:53.466138 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:53.466717 12297 external_mini_cluster.cc:1428] Setting key 68dc83390300c6ce846c2f923aa2ed90
I20260620 23:05:53.467419 19866 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:42931
WARNING: no policy specified for oryx/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab.
Entry for principal oryx/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab.
I20260620 23:05:53.507803 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37269
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab
--principal=oryx/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:53.809175 20077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:53.809438 20077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:53.809530 20077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:53.816515 20077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:53.816617 20077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:53.816696 20077 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:05:53.816772 20077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:05:53.826170 20077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab
--principal=oryx/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.2.126:37269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:53.827355 20077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:53.828585 20077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:53.840328 20085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:53.840410 20082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:53.840363 20083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:53.841236 20077 server_base.cc:1034] running on GCE node
I20260620 23:05:53.841800 20077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:53.842597 20077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:53.843814 20077 hybrid_clock.cc:648] HybridClock initialized: now 1781996753843745 us; error 40 us; skew 500 ppm
I20260620 23:05:53.844079 20077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:53 dist-test-slave-7g8g krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996753, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:53.846923 20077 init.cc:377] Logged in from keytab as oryx/127.12.2.66@KRBTEST.COM (short username oryx)
I20260620 23:05:53.848323 20077 webserver.cc:460] Webserver started at http://127.12.2.66:41115/ using document root <none> and password file <none>
I20260620 23:05:53.848973 20077 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:53.849058 20077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:53.849315 20077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:53.852085 20077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "8ded67fd82424dd6b1aa013e371cb694"
format_stamp: "Formatted at 2026-06-20 23:05:53 on dist-test-slave-7g8g"
server_key: "9f9459e0de7bb91f113780db0270e259"
server_key_iv: "ba310e8708c0900505e4e8062af73647"
server_key_version: "encryptionkey@0"
I20260620 23:05:53.852815 20077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "8ded67fd82424dd6b1aa013e371cb694"
format_stamp: "Formatted at 2026-06-20 23:05:53 on dist-test-slave-7g8g"
server_key: "9f9459e0de7bb91f113780db0270e259"
server_key_iv: "ba310e8708c0900505e4e8062af73647"
server_key_version: "encryptionkey@0"
I20260620 23:05:53.857911 20077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:05:53.861542 20092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:53.863050 20077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:53.863204 20077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "8ded67fd82424dd6b1aa013e371cb694"
format_stamp: "Formatted at 2026-06-20 23:05:53 on dist-test-slave-7g8g"
server_key: "9f9459e0de7bb91f113780db0270e259"
server_key_iv: "ba310e8708c0900505e4e8062af73647"
server_key_version: "encryptionkey@0"
I20260620 23:05:53.863415 20077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:53.875839 20077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:53.879923 20077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:53.880158 20077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:53.882563 20077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:53.882658 20077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:53.882778 20077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:53.882843 20077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:05:53 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996753, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:53.903780 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:53.883193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:60119 (local address 127.12.2.126:37269)
0620 23:05:53.883346 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:53.883350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:53.884791 (+ 1441us) server_negotiation.cc:407] Connection header received
0620 23:05:53.886705 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:53.886711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:53.886790 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:53.886875 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:53.888943 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.890064 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.892295 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.892821 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.892943 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:53.897370 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:53.897377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:53.897381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:53.897440 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:53.900324 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:53.901265 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:53.901272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:53.901276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:53.901441 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:53.901864 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:53.901869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:53.901873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:53.902257 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:53.902413 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:53.903138 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 23:05:53.903387 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:05:53.904719 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:53.883869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37269 (local address 127.12.2.66:60119)
0620 23:05:53.884640 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:53.884679 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:53.886416 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:53.887165 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:53.887177 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:53.887927 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:53.888728 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:53.888756 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.890228 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.890233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:53.892159 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:53.892170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.893075 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.893082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:53.893314 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:53.894075 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:53.894096 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:53.897125 (+ 3029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:53.900473 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:53.900485 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:53.900504 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:53.901103 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:53.901559 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:53.901563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:53.901565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:53.901760 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:53.902405 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:53.902410 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:53.902758 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 23:05:53.903043 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 23:05:53.903457 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":357,"threads_started":1}
I20260620 23:05:53.927423 20077 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:36613
I20260620 23:05:53.927466 20211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:36613 every 8 connection(s)
I20260620 23:05:53.928227 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:53.914298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:54373 (local address 127.12.2.126:37269)
0620 23:05:53.914441 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:53.914444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:53.915115 (+ 671us) server_negotiation.cc:407] Connection header received
0620 23:05:53.915368 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:53.915372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:53.915432 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:53.915509 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:53.916863 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.917822 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.919563 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.919994 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.920117 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:53.922463 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:53.922468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:53.922471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:53.922501 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:53.925493 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:53.926172 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:53.926178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:53.926182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:53.926318 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:53.926723 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:53.926729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:53.926732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:53.927082 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:53.927207 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:53.927705 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 23:05:53.927907 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:05:53.928285 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:53.914373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37269 (local address 127.12.2.66:54373)
0620 23:05:53.914995 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:53.915016 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:53.915232 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:53.915563 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:53.915567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:53.916224 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:53.916654 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:53.916671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.918002 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.918008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:53.919431 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:53.919446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:53.920221 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:53.920227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:53.920445 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:53.921202 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:53.921222 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:53.922287 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:53.925611 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:53.925616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:53.925619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:53.926070 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:53.926417 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:53.926422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:53.926427 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:53.926582 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:53.927189 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:53.927194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:53.927379 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 23:05:53.927527 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 23:05:53.927853 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":301,"threads_started":1}
I20260620 23:05:53.928942 20077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:53.931041 20212 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37269
I20260620 23:05:53.931322 20212 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:53.932031 20212 heartbeater.cc:511] Master 127.12.2.126:37269 requested a full tablet report, sending...
I20260620 23:05:53.933367 19865 ts_manager.cc:194] Registered new tserver with Master: 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66:36613)
I20260620 23:05:53.934315 19865 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:60119
I20260620 23:05:53.936367 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 20077
I20260620 23:05:53.936498 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:53.936920 12297 external_mini_cluster.cc:1428] Setting key b5be73caf45193353b1daaf1285ac873
WARNING: no policy specified for oryx/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab.
Entry for principal oryx/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab.
I20260620 23:05:53.979702 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37269
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab
--principal=oryx/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:54.292985 20216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:54.293273 20216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:54.293398 20216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:54.300657 20216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:54.300769 20216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:54.300894 20216 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:05:54.300998 20216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:05:54.310786 20216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab
--principal=oryx/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.2.126:37269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:54.312017 20216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:54.313297 20216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:54.325695 20222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:54.325810 20221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:54.325773 20224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:54.326606 20216 server_base.cc:1034] running on GCE node
I20260620 23:05:54.327268 20216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:54.328162 20216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:54.329375 20216 hybrid_clock.cc:648] HybridClock initialized: now 1781996754329294 us; error 53 us; skew 500 ppm
I20260620 23:05:54.329641 20216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:54 dist-test-slave-7g8g krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996754, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:54.332647 20216 init.cc:377] Logged in from keytab as oryx/127.12.2.67@KRBTEST.COM (short username oryx)
I20260620 23:05:54.333972 20216 webserver.cc:460] Webserver started at http://127.12.2.67:33185/ using document root <none> and password file <none>
I20260620 23:05:54.334621 20216 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:54.334710 20216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:54.334990 20216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:54.337805 20216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "741baf7e14d64762ba9449dcbb2e46d6"
format_stamp: "Formatted at 2026-06-20 23:05:54 on dist-test-slave-7g8g"
server_key: "0b19ed82f918cd25194eff2136cca5dc"
server_key_iv: "06964c4842b8007aaa42901eff63ca28"
server_key_version: "encryptionkey@0"
I20260620 23:05:54.338546 20216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "741baf7e14d64762ba9449dcbb2e46d6"
format_stamp: "Formatted at 2026-06-20 23:05:54 on dist-test-slave-7g8g"
server_key: "0b19ed82f918cd25194eff2136cca5dc"
server_key_iv: "06964c4842b8007aaa42901eff63ca28"
server_key_version: "encryptionkey@0"
I20260620 23:05:54.343528 20216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:05:54.346777 20231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:54.348157 20216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:05:54.348305 20216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "741baf7e14d64762ba9449dcbb2e46d6"
format_stamp: "Formatted at 2026-06-20 23:05:54 on dist-test-slave-7g8g"
server_key: "0b19ed82f918cd25194eff2136cca5dc"
server_key_iv: "06964c4842b8007aaa42901eff63ca28"
server_key_version: "encryptionkey@0"
I20260620 23:05:54.348496 20216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:54.373811 20216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:54.377660 20216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:54.377935 20216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:54.380651 20216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:54.380743 20216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:54.380874 20216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:54.380942 20216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:05:54 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996754, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:54.403735 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.381113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:37607 (local address 127.12.2.126:37269)
0620 23:05:54.381357 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:54.381364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:54.382786 (+ 1422us) server_negotiation.cc:407] Connection header received
0620 23:05:54.384751 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:54.384757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:54.384843 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:54.384937 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:54.387560 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.388438 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.391331 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.391888 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.392011 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.396263 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:54.396270 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:54.396275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:54.396315 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:54.399473 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:54.400332 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:54.400338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:54.400341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:54.400518 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:54.401118 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:54.401123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:54.401126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:54.401524 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:54.401662 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:54.403135 (+ 1473us) server_negotiation.cc:299] Negotiation successful
0620 23:05:54.403381 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:05:54.405081 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.381768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37269 (local address 127.12.2.67:37607)
0620 23:05:54.382599 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:54.382653 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:54.384482 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:54.385398 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:54.385415 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:54.386288 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:54.387348 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.387379 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.388604 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.388610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.391164 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.391182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.392255 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.392263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.392548 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.393276 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:54.393297 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:54.396052 (+ 2755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:54.399594 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:54.399603 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:54.399625 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:54.400203 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:54.400663 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:54.400668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:54.400673 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:54.400964 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:54.401742 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:54.401752 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:54.402320 (+ 568us) client_negotiation.cc:769] Sending connection context
0620 23:05:54.402677 (+ 357us) client_negotiation.cc:240] Negotiation successful
0620 23:05:54.403354 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":340,"threads_started":1}
I20260620 23:05:54.417227 20216 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:45787
I20260620 23:05:54.417264 20350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:45787 every 8 connection(s)
I20260620 23:05:54.418406 20216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:54.419565 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 20216
I20260620 23:05:54.419723 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:54.420475 20351 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37269
I20260620 23:05:54.420547 12297 external_mini_cluster.cc:1428] Setting key 2133c7a8d332e70f3364d50b1ce68ff6
I20260620 23:05:54.420784 20351 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:54.421666 20351 heartbeater.cc:511] Master 127.12.2.126:37269 requested a full tablet report, sending...
I20260620 23:05:54.423234 19866 ts_manager.cc:194] Registered new tserver with Master: 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787)
I20260620 23:05:54.424284 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.411454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37269 (local address 127.12.2.67:44217)
0620 23:05:54.412115 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:54.412132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:54.412308 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:54.412693 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:54.412699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:54.413096 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:54.413364 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.413372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.414695 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.414700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.416015 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.416026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.416701 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.416711 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.416871 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.417466 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:54.417485 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:54.418163 (+ 678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:54.421504 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:54.421509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:54.421513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:54.422015 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:54.422450 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:54.422454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:54.422458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:54.422601 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:54.423210 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:54.423215 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:54.423386 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 23:05:54.423540 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 23:05:54.423838 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":522,"threads_started":1}
I20260620 23:05:54.424281 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.411819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:44217 (local address 127.12.2.126:37269)
0620 23:05:54.411983 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:54.411988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:54.412204 (+ 216us) server_negotiation.cc:407] Connection header received
0620 23:05:54.412437 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:54.412442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:54.412534 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:54.412631 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:54.413480 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.414568 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.416141 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.416530 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.416630 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.418332 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:54.418336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:54.418340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:54.418367 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:54.421366 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:54.422167 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:54.422172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:54.422175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:54.422331 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:54.422740 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:54.422745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:54.422748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:54.423080 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:54.423224 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:54.423679 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 23:05:54.423897 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:05:54.424280 19866 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:37607
I20260620 23:05:54.435494 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:05:54 dist-test-slave-7g8g krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996754, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260620 23:05:54.469928 20073 heartbeater.cc:503] Master 127.12.2.126:37269 was elected leader, sending a full tablet report...
Jun 20 23:05:54 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:05:54 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:05:54.485864 19961 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:05:54.476394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.12.2.126:37269)
0620 23:05:54.476583 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:54.476589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:54.476699 (+ 110us) server_negotiation.cc:407] Connection header received
0620 23:05:54.476885 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:54.476889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:54.476949 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:54.477028 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:54.478359 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.479126 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.480331 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.480718 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.480825 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.485659 (+ 4834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41650: BlockingRecv error: recv got EOF from 127.0.0.1:41650 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 23:05:54 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996754, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:54.501466 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.489094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.12.2.126:37269)
0620 23:05:54.489303 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:54.489309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:54.489538 (+ 229us) server_negotiation.cc:407] Connection header received
0620 23:05:54.489699 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:54.489705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:54.489780 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:54.489859 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:54.490995 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.491713 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.492894 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.493313 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.493415 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.496910 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:54.496915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:54.496918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:54.496950 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:54.499132 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:54.499728 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:54.499734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:54.499737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:54.499847 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:54.500170 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:54.500182 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:54.500185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:54.500452 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:54.500694 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:54.500955 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 23:05:54.501161 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:05:54.505122 19866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:54.511857 19866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:54.546507 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.523750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.12.2.65:43785)
0620 23:05:54.539874 (+ 16124us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:54.539882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:54.539909 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:05:54.540004 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:54.540012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:54.540242 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:54.540414 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:54.541013 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.542183 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.544267 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.545504 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.545615 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.545824 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:54.546066 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 23:05:54.546225 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15978,"thread_start_us":15848,"threads_started":1}
I20260620 23:05:54.546593 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.523557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:43785 (local address 127.0.0.1:35438)
0620 23:05:54.534041 (+ 10484us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:54.534090 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:54.534291 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:54.540583 (+ 6292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:54.540586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:54.540603 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:54.540869 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.540876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.542363 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.542368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.544121 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.544132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.545672 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.545678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.545855 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.545901 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:05:54.546005 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:05:54.546181 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10197,"mutex_wait_us":21,"thread_start_us":16421,"threads_started":1}
I20260620 23:05:54.547776 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.524382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.12.2.66:36613)
0620 23:05:54.539486 (+ 15104us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:54.539494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:54.540523 (+ 1029us) server_negotiation.cc:407] Connection header received
0620 23:05:54.540621 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:54.540629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:54.540857 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:54.541038 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:54.541532 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.542798 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.545214 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.546496 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.546602 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.546776 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:54.547377 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 23:05:54.547539 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14952,"thread_start_us":14958,"threads_started":1}
I20260620 23:05:54.547857 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.523990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:45787 (local address 127.0.0.1:51220)
0620 23:05:54.537825 (+ 13835us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:54.537843 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:54.537963 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:54.539595 (+ 1632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:54.539603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:54.539633 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:54.540075 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.540087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.542069 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.542074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.544531 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.544544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.546571 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.546950 (+ 379us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.547152 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.547235 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 23:05:54.547317 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 23:05:54.547496 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13711,"mutex_wait_us":87,"thread_start_us":9646,"threads_started":1}
I20260620 23:05:54.548414 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.524567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.12.2.67:45787)
0620 23:05:54.535485 (+ 10918us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:54.535496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:54.538078 (+ 2582us) server_negotiation.cc:407] Connection header received
0620 23:05:54.538187 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:54.538196 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:54.538497 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:54.538678 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:54.540378 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.541934 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.544703 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.546088 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.546202 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.546410 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:54.548022 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0620 23:05:54.548178 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10202,"thread_start_us":10022,"threads_started":1}
I20260620 23:05:54.550047 20003 tablet_service.cc:1467] Processing CreateTablet for tablet 6978333c7cdb423994481a96f94f0dbd (DEFAULT_TABLE table=test-table [id=1c68a289b808498c9f6a265d589bf103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:54.550951 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.524040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:36613 (local address 127.0.0.1:33612)
0620 23:05:54.540260 (+ 16220us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:54.540274 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:54.540388 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:54.541023 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:54.541027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:54.541049 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:54.541364 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.541376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.542964 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.542970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.545051 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.545063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.546895 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.546901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.547151 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.547237 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 23:05:54.547320 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:05:54.547492 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16102,"mutex_wait_us":114,"thread_start_us":14907,"threads_started":1}
I20260620 23:05:54.551324 20003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6978333c7cdb423994481a96f94f0dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:54.551281 20281 tablet_service.cc:1467] Processing CreateTablet for tablet 6978333c7cdb423994481a96f94f0dbd (DEFAULT_TABLE table=test-table [id=1c68a289b808498c9f6a265d589bf103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:54.552767 20281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6978333c7cdb423994481a96f94f0dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:54.554104 20142 tablet_service.cc:1467] Processing CreateTablet for tablet 6978333c7cdb423994481a96f94f0dbd (DEFAULT_TABLE table=test-table [id=1c68a289b808498c9f6a265d589bf103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:54.555328 20142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6978333c7cdb423994481a96f94f0dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:54.563607 20375 tablet_bootstrap.cc:492] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: Bootstrap starting.
I20260620 23:05:54.566530 20375 tablet_bootstrap.cc:654] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:54.566597 20376 tablet_bootstrap.cc:492] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb: Bootstrap starting.
I20260620 23:05:54.567993 20375 log.cc:824] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:54.568989 20377 tablet_bootstrap.cc:492] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694: Bootstrap starting.
I20260620 23:05:54.570066 20376 tablet_bootstrap.cc:654] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:54.571332 20376 log.cc:824] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:54.571775 20377 tablet_bootstrap.cc:654] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:54.572888 20377 log.cc:824] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:54.574074 20375 tablet_bootstrap.cc:492] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: No bootstrap required, opened a new log
I20260620 23:05:54.574098 20376 tablet_bootstrap.cc:492] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb: No bootstrap required, opened a new log
I20260620 23:05:54.574316 20376 ts_tablet_manager.cc:1397] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 23:05:54.574328 20375 ts_tablet_manager.cc:1397] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260620 23:05:54.575510 20377 tablet_bootstrap.cc:492] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694: No bootstrap required, opened a new log
I20260620 23:05:54.575733 20377 ts_tablet_manager.cc:1397] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 23:05:54.581065 20376 raft_consensus.cc:348] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } }
I20260620 23:05:54.581393 20376 raft_consensus.cc:374] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:54.581481 20376 raft_consensus.cc:729] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd320521b0a45949654b6ad5163b6cb, State: Initialized, Role: FOLLOWER
I20260620 23:05:54.582029 20376 consensus_queue.cc:260] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } }
I20260620 23:05:54.583628 20376 ts_tablet_manager.cc:1428] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20260620 23:05:54.584928 20375 raft_consensus.cc:348] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } }
I20260620 23:05:54.585060 20377 raft_consensus.cc:348] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } }
I20260620 23:05:54.585350 20375 raft_consensus.cc:374] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:54.585354 20377 raft_consensus.cc:374] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:54.585443 20375 raft_consensus.cc:729] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741baf7e14d64762ba9449dcbb2e46d6, State: Initialized, Role: FOLLOWER
I20260620 23:05:54.585443 20377 raft_consensus.cc:729] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ded67fd82424dd6b1aa013e371cb694, State: Initialized, Role: FOLLOWER
I20260620 23:05:54.585947 20377 consensus_queue.cc:260] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } }
I20260620 23:05:54.586055 20375 consensus_queue.cc:260] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } }
I20260620 23:05:54.586959 20212 heartbeater.cc:503] Master 127.12.2.126:37269 was elected leader, sending a full tablet report...
I20260620 23:05:54.587677 20351 heartbeater.cc:503] Master 127.12.2.126:37269 was elected leader, sending a full tablet report...
I20260620 23:05:54.588277 20375 ts_tablet_manager.cc:1428] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 23:05:54.589645 20377 ts_tablet_manager.cc:1428] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 23:05:54.616282 20382 raft_consensus.cc:482] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:54.616585 20382 raft_consensus.cc:504] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } }
I20260620 23:05:54.618208 20382 leader_election.cc:288] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65:43785), 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787)
I20260620 23:05:54.624682 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.618802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:43897 (local address 127.12.2.67:45787)
0620 23:05:54.619022 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:54.619028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:54.619447 (+ 419us) server_negotiation.cc:407] Connection header received
0620 23:05:54.619722 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:54.619727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:54.619811 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:54.619868 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:54.620621 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.621298 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.623028 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.623816 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.623906 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.623992 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:54.624314 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 23:05:54.624436 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:05:54.624691 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.618624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:47317 (local address 127.12.2.65:43785)
0620 23:05:54.618832 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:54.618838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:54.619645 (+ 807us) server_negotiation.cc:407] Connection header received
0620 23:05:54.619824 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:54.619830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:54.619899 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:54.619977 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:54.620524 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.621227 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.623094 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.623878 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.624015 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.624112 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:54.624359 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 23:05:54.624480 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:05:54.625036 20384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.618445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:43785 (local address 127.12.2.66:47317)
0620 23:05:54.619536 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:54.619550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:54.619687 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:54.619999 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:54.620002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:54.620020 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:54.620348 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.620358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.621360 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.621365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.622923 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.622961 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.624082 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.624087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.624213 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.624235 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:05:54.624318 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:05:54.624480 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":242,"thread_start_us":383,"threads_started":1}
I20260620 23:05:54.625041 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.618445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:45787 (local address 127.12.2.66:43897)
0620 23:05:54.619346 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:54.619368 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:54.619560 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:54.620056 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:54.620060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:54.620150 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:54.620500 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.620510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.621424 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.621428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.622883 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:54.622894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.624008 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.624014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:54.624177 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.624200 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:05:54.624276 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:05:54.624389 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":2,"spinlock_wait_cycles":30592,"thread_start_us":730,"threads_started":1}
I20260620 23:05:54.625888 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978333c7cdb423994481a96f94f0dbd" candidate_uuid: "8ded67fd82424dd6b1aa013e371cb694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd320521b0a45949654b6ad5163b6cb" is_pre_election: true
I20260620 23:05:54.625905 20301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978333c7cdb423994481a96f94f0dbd" candidate_uuid: "8ded67fd82424dd6b1aa013e371cb694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741baf7e14d64762ba9449dcbb2e46d6" is_pre_election: true
I20260620 23:05:54.626335 20028 raft_consensus.cc:2454] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ded67fd82424dd6b1aa013e371cb694 in term 0.
I20260620 23:05:54.626344 20301 raft_consensus.cc:2454] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ded67fd82424dd6b1aa013e371cb694 in term 0.
I20260620 23:05:54.626910 20094 leader_election.cc:302] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd320521b0a45949654b6ad5163b6cb, 8ded67fd82424dd6b1aa013e371cb694; no voters:
I20260620 23:05:54.627316 20382 raft_consensus.cc:2781] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:54.627444 20382 raft_consensus.cc:482] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:54.627532 20382 raft_consensus.cc:3037] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:54.629596 20382 raft_consensus.cc:504] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } }
I20260620 23:05:54.630107 20382 leader_election.cc:288] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [CANDIDATE]: Term 1 election: Requested vote from peers 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65:43785), 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787)
I20260620 23:05:54.630478 20301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978333c7cdb423994481a96f94f0dbd" candidate_uuid: "8ded67fd82424dd6b1aa013e371cb694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741baf7e14d64762ba9449dcbb2e46d6"
I20260620 23:05:54.630481 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978333c7cdb423994481a96f94f0dbd" candidate_uuid: "8ded67fd82424dd6b1aa013e371cb694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd320521b0a45949654b6ad5163b6cb"
I20260620 23:05:54.630630 20301 raft_consensus.cc:3037] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:54.630636 20028 raft_consensus.cc:3037] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:54.632666 20028 raft_consensus.cc:2454] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ded67fd82424dd6b1aa013e371cb694 in term 1.
I20260620 23:05:54.632722 20301 raft_consensus.cc:2454] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ded67fd82424dd6b1aa013e371cb694 in term 1.
I20260620 23:05:54.633071 20093 leader_election.cc:302] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741baf7e14d64762ba9449dcbb2e46d6, 8ded67fd82424dd6b1aa013e371cb694; no voters:
I20260620 23:05:54.633353 20382 raft_consensus.cc:2781] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:54.633777 20382 raft_consensus.cc:686] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 1 LEADER]: Becoming Leader. State: Replica: 8ded67fd82424dd6b1aa013e371cb694, State: Running, Role: LEADER
I20260620 23:05:54.634233 20382 consensus_queue.cc:237] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } }
I20260620 23:05:54.638371 19866 catalog_manager.cc:5495] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66). New cstate: current_term: 1 leader_uuid: "8ded67fd82424dd6b1aa013e371cb694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } health_report { overall_health: HEALTHY } } }
I20260620 23:05:54.654573 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:54.649389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.12.2.66:36613)
0620 23:05:54.649569 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:54.649575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:54.649594 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:05:54.649816 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:54.649821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:54.649902 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:54.650025 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:54.650370 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.651128 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.652391 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:54.652781 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:54.652885 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:54.653490 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:54.654024 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:54.654152 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:54.654253 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 23:05:54.654332 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:05:54.662207 20028 raft_consensus.cc:1261] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Refusing update from remote peer 8ded67fd82424dd6b1aa013e371cb694: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:54.662209 20301 raft_consensus.cc:1261] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Refusing update from remote peer 8ded67fd82424dd6b1aa013e371cb694: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:54.663115 20386 consensus_queue.cc:1035] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:54.663445 20382 consensus_queue.cc:1035] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [LEADER]: Connected to new peer: Peer: permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:05:54.674108 20352 tablet.cc:2369] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:54.674840 20389 mvcc.cc:205] Tried to move back new op lower bound from 7299058707087794176 to 7299058706985410560. Current Snapshot: MvccSnapshot[applied={T|T < 7299058707087794176}]
I20260620 23:05:54.675390 20390 mvcc.cc:205] Tried to move back new op lower bound from 7299058707087794176 to 7299058706985410560. Current Snapshot: MvccSnapshot[applied={T|T < 7299058707087794176}]
I20260620 23:05:54.693804 19865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:54.694074 19865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:54.697772 19865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5: Sending DeleteTablet for 3 replicas of tablet 6978333c7cdb423994481a96f94f0dbd
I20260620 23:05:54.698911 20003 tablet_service.cc:1514] Processing DeleteTablet for tablet 6978333c7cdb423994481a96f94f0dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:54 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:35438
I20260620 23:05:54.698911 20281 tablet_service.cc:1514] Processing DeleteTablet for tablet 6978333c7cdb423994481a96f94f0dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:54 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:51220
I20260620 23:05:54.699447 20142 tablet_service.cc:1514] Processing DeleteTablet for tablet 6978333c7cdb423994481a96f94f0dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:54 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:33612
I20260620 23:05:54.701534 19865 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:54.702160 19865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:54.711404 20401 tablet_replica.cc:330] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694: stopping tablet replica
I20260620 23:05:54.711874 20401 raft_consensus.cc:2229] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:54.712036 20001 tablet_service.cc:1467] Processing CreateTablet for tablet ab2d645b4a1e4ba0a5f0be87d73b0577 (DEFAULT_TABLE table=test-table [id=d45c6be6355f43d58f6bc2980ca2ea66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:54.712031 20280 tablet_service.cc:1467] Processing CreateTablet for tablet ab2d645b4a1e4ba0a5f0be87d73b0577 (DEFAULT_TABLE table=test-table [id=d45c6be6355f43d58f6bc2980ca2ea66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:54.712564 20280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2d645b4a1e4ba0a5f0be87d73b0577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:54.712584 20001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2d645b4a1e4ba0a5f0be87d73b0577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:54.712548 20401 raft_consensus.cc:2258] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:54.717581 20401 ts_tablet_manager.cc:1905] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:54.719439 20375 tablet_bootstrap.cc:492] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6: Bootstrap starting.
I20260620 23:05:54.719969 20141 tablet_service.cc:1467] Processing CreateTablet for tablet ab2d645b4a1e4ba0a5f0be87d73b0577 (DEFAULT_TABLE table=test-table [id=d45c6be6355f43d58f6bc2980ca2ea66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:54.720445 20141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2d645b4a1e4ba0a5f0be87d73b0577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:54.720980 20375 tablet_bootstrap.cc:654] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:54.722273 20376 tablet_bootstrap.cc:492] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb: Bootstrap starting.
I20260620 23:05:54.724069 20376 tablet_bootstrap.cc:654] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:54.726688 20377 tablet_bootstrap.cc:492] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694: Bootstrap starting.
I20260620 23:05:54.728416 20377 tablet_bootstrap.cc:654] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:54.728813 20401 ts_tablet_manager.cc:1918] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:54.728996 20401 log.cc:1196] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/6978333c7cdb423994481a96f94f0dbd
I20260620 23:05:54.729514 20401 ts_tablet_manager.cc:1939] T 6978333c7cdb423994481a96f94f0dbd P 8ded67fd82424dd6b1aa013e371cb694: Deleting consensus metadata
I20260620 23:05:54.731262 19852 catalog_manager.cc:4841] TS 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66:36613): tablet 6978333c7cdb423994481a96f94f0dbd (table test-table [id=1c68a289b808498c9f6a265d589bf103]) successfully deleted
I20260620 23:05:54.736068 20377 tablet_bootstrap.cc:492] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694: No bootstrap required, opened a new log
I20260620 23:05:54.736258 20377 ts_tablet_manager.cc:1397] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 23:05:54.737058 20377 raft_consensus.cc:348] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:54.737296 20377 raft_consensus.cc:374] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:54.737387 20377 raft_consensus.cc:729] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ded67fd82424dd6b1aa013e371cb694, State: Initialized, Role: FOLLOWER
I20260620 23:05:54.737684 20377 consensus_queue.cc:260] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:54.738341 20377 ts_tablet_manager.cc:1428] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:54.743743 20400 tablet_replica.cc:330] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: stopping tablet replica
I20260620 23:05:54.744287 20400 raft_consensus.cc:2229] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:54.745121 20376 tablet_bootstrap.cc:492] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb: No bootstrap required, opened a new log
I20260620 23:05:54.745270 20376 ts_tablet_manager.cc:1397] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.006s
I20260620 23:05:54.745926 20376 raft_consensus.cc:348] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:54.746093 20376 raft_consensus.cc:374] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:54.746182 20376 raft_consensus.cc:729] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd320521b0a45949654b6ad5163b6cb, State: Initialized, Role: FOLLOWER
I20260620 23:05:54.746433 20376 consensus_queue.cc:260] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:54.745114 20375 tablet_bootstrap.cc:492] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6: No bootstrap required, opened a new log
I20260620 23:05:54.746791 20375 ts_tablet_manager.cc:1397] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.000s
I20260620 23:05:54.747092 20400 raft_consensus.cc:2258] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:54.747673 20375 raft_consensus.cc:348] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:54.747929 20375 raft_consensus.cc:374] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:54.748060 20375 raft_consensus.cc:729] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741baf7e14d64762ba9449dcbb2e46d6, State: Initialized, Role: FOLLOWER
I20260620 23:05:54.748409 20375 consensus_queue.cc:260] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:54.749505 20375 ts_tablet_manager.cc:1428] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:54.750697 20376 ts_tablet_manager.cc:1428] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260620 23:05:54.753198 20400 ts_tablet_manager.cc:1905] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:54.757992 20400 ts_tablet_manager.cc:1918] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:54.758127 20400 log.cc:1196] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/6978333c7cdb423994481a96f94f0dbd
I20260620 23:05:54.758803 20399 tablet_replica.cc:330] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb: stopping tablet replica
I20260620 23:05:54.759186 20400 ts_tablet_manager.cc:1939] T 6978333c7cdb423994481a96f94f0dbd P 741baf7e14d64762ba9449dcbb2e46d6: Deleting consensus metadata
I20260620 23:05:54.759397 20399 raft_consensus.cc:2229] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:54.759893 20399 raft_consensus.cc:2258] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:54.761087 19850 catalog_manager.cc:4841] TS 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787): tablet 6978333c7cdb423994481a96f94f0dbd (table test-table [id=1c68a289b808498c9f6a265d589bf103]) successfully deleted
I20260620 23:05:54.761536 20399 ts_tablet_manager.cc:1905] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:54.766141 20399 ts_tablet_manager.cc:1918] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:54.766278 20399 log.cc:1196] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/6978333c7cdb423994481a96f94f0dbd
I20260620 23:05:54.766712 20399 ts_tablet_manager.cc:1939] T 6978333c7cdb423994481a96f94f0dbd P 1dd320521b0a45949654b6ad5163b6cb: Deleting consensus metadata
I20260620 23:05:54.768069 19851 catalog_manager.cc:4841] TS 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65:43785): tablet 6978333c7cdb423994481a96f94f0dbd (table test-table [id=1c68a289b808498c9f6a265d589bf103]) successfully deleted
W20260620 23:05:54.931675 20213 tablet.cc:2369] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:05:54.964937 20074 tablet.cc:2369] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:55.041432 20383 raft_consensus.cc:482] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:55.041638 20383 raft_consensus.cc:504] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.043254 20383 leader_election.cc:288] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65:43785), 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66:36613)
I20260620 23:05:55.050860 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.043555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:45187 (local address 127.12.2.65:43785)
0620 23:05:55.043761 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.043767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.044800 (+ 1033us) server_negotiation.cc:407] Connection header received
0620 23:05:55.044891 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.044897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.044991 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.045097 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:55.045571 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.046643 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.049199 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.050003 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.050112 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.050210 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.050480 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.050604 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:05:55.050973 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.043724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:43785 (local address 127.12.2.67:45187)
0620 23:05:55.044689 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:55.044704 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:55.044793 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:55.045122 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:55.045126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:55.045150 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:55.045391 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.045403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.046761 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.046766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.049010 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.049021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.050145 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.050152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.050340 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.050364 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:05:55.050444 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:05:55.050570 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":94,"thread_start_us":500,"threads_started":1}
I20260620 23:05:55.051692 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.043936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:36299 (local address 127.12.2.66:36613)
0620 23:05:55.044117 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.044123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.044610 (+ 487us) server_negotiation.cc:407] Connection header received
0620 23:05:55.044894 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.044899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.044959 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.045008 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:55.045799 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.046574 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.049170 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.050454 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.050590 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.050680 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.051271 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.051406 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:05:55.051820 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2d645b4a1e4ba0a5f0be87d73b0577" candidate_uuid: "741baf7e14d64762ba9449dcbb2e46d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd320521b0a45949654b6ad5163b6cb" is_pre_election: true
I20260620 23:05:55.052064 20028 raft_consensus.cc:2454] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741baf7e14d64762ba9449dcbb2e46d6 in term 0.
I20260620 23:05:55.052384 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.043469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:36613 (local address 127.12.2.67:36299)
0620 23:05:55.044497 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:55.044517 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:55.044733 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:55.045180 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:55.045184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:55.045216 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:55.045639 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.045651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.046690 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.046708 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.049007 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.049021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.050845 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.050852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.051087 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.051122 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 23:05:55.051893 (+ 771us) client_negotiation.cc:240] Negotiation successful
0620 23:05:55.052033 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":121,"thread_start_us":1021,"threads_started":1}
I20260620 23:05:55.052645 20233 leader_election.cc:302] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd320521b0a45949654b6ad5163b6cb, 741baf7e14d64762ba9449dcbb2e46d6; no voters:
I20260620 23:05:55.053100 20383 raft_consensus.cc:2781] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:55.053228 20383 raft_consensus.cc:482] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:55.053335 20383 raft_consensus.cc:3037] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.053215 20166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2d645b4a1e4ba0a5f0be87d73b0577" candidate_uuid: "741baf7e14d64762ba9449dcbb2e46d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ded67fd82424dd6b1aa013e371cb694" is_pre_election: true
I20260620 23:05:55.053637 20166 raft_consensus.cc:2454] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741baf7e14d64762ba9449dcbb2e46d6 in term 0.
I20260620 23:05:55.056593 20383 raft_consensus.cc:504] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.057435 20383 leader_election.cc:288] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [CANDIDATE]: Term 1 election: Requested vote from peers 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65:43785), 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66:36613)
I20260620 23:05:55.057735 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2d645b4a1e4ba0a5f0be87d73b0577" candidate_uuid: "741baf7e14d64762ba9449dcbb2e46d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd320521b0a45949654b6ad5163b6cb"
I20260620 23:05:55.057909 20028 raft_consensus.cc:3037] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.059931 20028 raft_consensus.cc:2454] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741baf7e14d64762ba9449dcbb2e46d6 in term 1.
I20260620 23:05:55.060321 20233 leader_election.cc:302] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd320521b0a45949654b6ad5163b6cb, 741baf7e14d64762ba9449dcbb2e46d6; no voters:
I20260620 23:05:55.060599 20383 raft_consensus.cc:2781] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:55.060614 20166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2d645b4a1e4ba0a5f0be87d73b0577" candidate_uuid: "741baf7e14d64762ba9449dcbb2e46d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ded67fd82424dd6b1aa013e371cb694"
I20260620 23:05:55.060848 20166 raft_consensus.cc:3037] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.061023 20383 raft_consensus.cc:686] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 LEADER]: Becoming Leader. State: Replica: 741baf7e14d64762ba9449dcbb2e46d6, State: Running, Role: LEADER
I20260620 23:05:55.061429 20383 consensus_queue.cc:237] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.064368 20166 raft_consensus.cc:2454] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741baf7e14d64762ba9449dcbb2e46d6 in term 1.
I20260620 23:05:55.064680 19866 catalog_manager.cc:5495] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67). New cstate: current_term: 1 leader_uuid: "741baf7e14d64762ba9449dcbb2e46d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } health_report { overall_health: HEALTHY } } }
Jun 20 23:05:55 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996752, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.126@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:55.126210 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.113120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37269 (local address 127.0.0.1:41666)
0620 23:05:55.113737 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:55.113759 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:55.113920 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:55.114306 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:55.114310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:55.114962 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:55.115133 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.115142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.116150 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.116154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.117079 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.117090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.117742 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.117747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.117891 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.118586 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:55.118602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:55.120985 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:55.123840 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:55.123850 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:55.123856 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:55.124289 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:55.124632 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:55.124635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:55.124638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:55.124806 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:55.125343 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:55.125351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:55.125529 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 23:05:55.125671 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 23:05:55.125897 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":327,"threads_started":1}
I20260620 23:05:55.126324 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.113180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.12.2.126:37269)
0620 23:05:55.114040 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.114044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.114059 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:05:55.114095 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.114098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.114156 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.114243 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:55.115273 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.116025 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.117200 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.117599 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.117721 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.121188 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:55.121193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:55.121196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:55.121225 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:55.123696 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:55.124411 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:55.124415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:55.124418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:55.124541 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:55.124886 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:55.124889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:55.124891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:55.125211 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:55.125313 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.125784 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.125992 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":747,"thread_start_us":558,"threads_started":1}
I20260620 23:05:55.131070 19866 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:41666:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:05:55.132097 19866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:55.138789 20141 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4adcc562b94ec2a6cd19d89d3dfafe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1accc640f2c243af9428274be3039986]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:05:55.139328 20141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4adcc562b94ec2a6cd19d89d3dfafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:55.139276 20281 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4adcc562b94ec2a6cd19d89d3dfafe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1accc640f2c243af9428274be3039986]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:05:55.139789 20281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4adcc562b94ec2a6cd19d89d3dfafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:55.144533 20377 tablet_bootstrap.cc:492] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: Bootstrap starting.
I20260620 23:05:55.144948 20003 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4adcc562b94ec2a6cd19d89d3dfafe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1accc640f2c243af9428274be3039986]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:05:55.145390 20003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4adcc562b94ec2a6cd19d89d3dfafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:55.145633 20375 tablet_bootstrap.cc:492] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6: Bootstrap starting.
I20260620 23:05:55.146126 20377 tablet_bootstrap.cc:654] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:55.147143 20375 tablet_bootstrap.cc:654] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:55.148018 20377 tablet_bootstrap.cc:492] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: No bootstrap required, opened a new log
I20260620 23:05:55.148159 20377 ts_tablet_manager.cc:1397] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:05:55.148694 20377 raft_consensus.cc:348] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } }
I20260620 23:05:55.148888 20377 raft_consensus.cc:374] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:55.148962 20377 raft_consensus.cc:729] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ded67fd82424dd6b1aa013e371cb694, State: Initialized, Role: FOLLOWER
I20260620 23:05:55.149405 20377 consensus_queue.cc:260] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } }
I20260620 23:05:55.149819 20382 tablet_replica.cc:439] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } }
I20260620 23:05:55.150043 20382 tablet_replica.cc:442] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:55.150817 20377 ts_tablet_manager.cc:1428] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:55.152256 20375 tablet_bootstrap.cc:492] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6: No bootstrap required, opened a new log
I20260620 23:05:55.152413 20375 ts_tablet_manager.cc:1397] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 23:05:55.152511 20376 tablet_bootstrap.cc:492] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb: Bootstrap starting.
I20260620 23:05:55.153106 20375 raft_consensus.cc:348] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } }
I20260620 23:05:55.153354 20375 raft_consensus.cc:374] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:55.153430 20375 raft_consensus.cc:729] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741baf7e14d64762ba9449dcbb2e46d6, State: Initialized, Role: FOLLOWER
I20260620 23:05:55.153646 20375 consensus_queue.cc:260] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } }
I20260620 23:05:55.154062 20376 tablet_bootstrap.cc:654] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:55.154181 20375 ts_tablet_manager.cc:1428] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:05:55.154057 20383 tablet_replica.cc:439] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } }
I20260620 23:05:55.154354 20383 tablet_replica.cc:442] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:55.156623 20376 tablet_bootstrap.cc:492] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb: No bootstrap required, opened a new log
I20260620 23:05:55.156796 20376 ts_tablet_manager.cc:1397] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 23:05:55.157444 20376 raft_consensus.cc:348] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } }
I20260620 23:05:55.157629 20376 raft_consensus.cc:374] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:55.157713 20376 raft_consensus.cc:729] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd320521b0a45949654b6ad5163b6cb, State: Initialized, Role: FOLLOWER
I20260620 23:05:55.157927 20376 consensus_queue.cc:260] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } }
I20260620 23:05:55.158440 20376 ts_tablet_manager.cc:1428] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:05:55.158284 20381 tablet_replica.cc:439] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } }
I20260620 23:05:55.158574 20381 tablet_replica.cc:442] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:55.181423 20382 raft_consensus.cc:482] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:55.181589 20382 raft_consensus.cc:504] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } }
I20260620 23:05:55.182201 20382 leader_election.cc:288] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787), 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65:43785)
I20260620 23:05:55.182662 20301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4adcc562b94ec2a6cd19d89d3dfafe" candidate_uuid: "8ded67fd82424dd6b1aa013e371cb694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741baf7e14d64762ba9449dcbb2e46d6" is_pre_election: true
I20260620 23:05:55.182684 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4adcc562b94ec2a6cd19d89d3dfafe" candidate_uuid: "8ded67fd82424dd6b1aa013e371cb694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd320521b0a45949654b6ad5163b6cb" is_pre_election: true
I20260620 23:05:55.182916 20301 raft_consensus.cc:2454] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ded67fd82424dd6b1aa013e371cb694 in term 0.
I20260620 23:05:55.182955 20028 raft_consensus.cc:2454] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ded67fd82424dd6b1aa013e371cb694 in term 0.
I20260620 23:05:55.183305 20093 leader_election.cc:302] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741baf7e14d64762ba9449dcbb2e46d6, 8ded67fd82424dd6b1aa013e371cb694; no voters:
I20260620 23:05:55.183586 20394 raft_consensus.cc:2781] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:55.183684 20394 raft_consensus.cc:482] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:55.183768 20394 raft_consensus.cc:3037] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.185631 20394 raft_consensus.cc:504] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } }
I20260620 23:05:55.186213 20394 leader_election.cc:288] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [CANDIDATE]: Term 1 election: Requested vote from peers 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787), 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65:43785)
I20260620 23:05:55.186625 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4adcc562b94ec2a6cd19d89d3dfafe" candidate_uuid: "8ded67fd82424dd6b1aa013e371cb694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd320521b0a45949654b6ad5163b6cb"
I20260620 23:05:55.186620 20301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4adcc562b94ec2a6cd19d89d3dfafe" candidate_uuid: "8ded67fd82424dd6b1aa013e371cb694" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741baf7e14d64762ba9449dcbb2e46d6"
I20260620 23:05:55.186793 20301 raft_consensus.cc:3037] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.186811 20028 raft_consensus.cc:3037] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.188946 20301 raft_consensus.cc:2454] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ded67fd82424dd6b1aa013e371cb694 in term 1.
I20260620 23:05:55.188931 20028 raft_consensus.cc:2454] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ded67fd82424dd6b1aa013e371cb694 in term 1.
I20260620 23:05:55.189291 20093 leader_election.cc:302] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741baf7e14d64762ba9449dcbb2e46d6, 8ded67fd82424dd6b1aa013e371cb694; no voters:
I20260620 23:05:55.189533 20394 raft_consensus.cc:2781] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:55.189700 20394 raft_consensus.cc:686] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [term 1 LEADER]: Becoming Leader. State: Replica: 8ded67fd82424dd6b1aa013e371cb694, State: Running, Role: LEADER
I20260620 23:05:55.189940 20394 consensus_queue.cc:237] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } }
I20260620 23:05:55.191298 20382 tablet_replica.cc:439] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: TxnStatusTablet state changed. Reason: New leader 8ded67fd82424dd6b1aa013e371cb694. Latest consensus state: current_term: 1 leader_uuid: "8ded67fd82424dd6b1aa013e371cb694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } }
I20260620 23:05:55.191537 20382 tablet_replica.cc:442] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:05:55.192085 20420 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:05:55.192294 20420 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:05:55.193063 19866 catalog_manager.cc:5495] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66). New cstate: current_term: 1 leader_uuid: "8ded67fd82424dd6b1aa013e371cb694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:55.214562 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.208978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:36613 (local address 127.0.0.1:33624)
0620 23:05:55.209212 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:55.209232 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:55.209368 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:55.209993 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:55.209996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:55.210007 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:55.210150 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.210157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.211622 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.211626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.212626 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.212637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.213544 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.213548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.213701 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.213764 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:55.214156 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:55.214177 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:05:55.214238 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:05:55.214347 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 23:05:55.214635 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.209068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.12.2.66:36613)
0620 23:05:55.209647 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.209653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.209674 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:05:55.209761 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.209778 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.209857 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.209939 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:55.210274 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.211471 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.212756 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.213408 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.213541 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.213932 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:55.214061 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:55.214148 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.214285 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.214366 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462}
I20260620 23:05:55.218276 20301 raft_consensus.cc:1261] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Refusing update from remote peer 8ded67fd82424dd6b1aa013e371cb694: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:55.218321 20028 raft_consensus.cc:1261] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Refusing update from remote peer 8ded67fd82424dd6b1aa013e371cb694: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:55.218789 20382 consensus_queue.cc:1035] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [LEADER]: Connected to new peer: Peer: permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:55.219076 20394 consensus_queue.cc:1035] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:55.224766 20381 tablet_replica.cc:439] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb: TxnStatusTablet state changed. Reason: New leader 8ded67fd82424dd6b1aa013e371cb694. Latest consensus state: current_term: 1 leader_uuid: "8ded67fd82424dd6b1aa013e371cb694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } }
I20260620 23:05:55.225102 20381 tablet_replica.cc:442] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:55.227217 20394 tablet_replica.cc:439] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ded67fd82424dd6b1aa013e371cb694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } }
I20260620 23:05:55.227526 20394 tablet_replica.cc:442] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:05:55.231015 20381 tablet_replica.cc:439] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ded67fd82424dd6b1aa013e371cb694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } }
I20260620 23:05:55.231256 20381 tablet_replica.cc:442] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 1dd320521b0a45949654b6ad5163b6cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:55.237804 20383 tablet_replica.cc:439] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6: TxnStatusTablet state changed. Reason: New leader 8ded67fd82424dd6b1aa013e371cb694. Latest consensus state: current_term: 1 leader_uuid: "8ded67fd82424dd6b1aa013e371cb694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } }
I20260620 23:05:55.238123 20383 tablet_replica.cc:442] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:55.239251 20382 tablet_replica.cc:439] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ded67fd82424dd6b1aa013e371cb694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } }
I20260620 23:05:55.239497 20382 tablet_replica.cc:442] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 8ded67fd82424dd6b1aa013e371cb694: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:05:55.245427 20424 mvcc.cc:205] Tried to move back new op lower bound from 7299058709368782848 to 7299058709260177408. Current Snapshot: MvccSnapshot[applied={T|T < 7299058709368782848}]
I20260620 23:05:55.254899 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.236980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.12.2.67:45787)
0620 23:05:55.238122 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.238128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.247470 (+ 9342us) server_negotiation.cc:407] Connection header received
0620 23:05:55.247676 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.247681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.247800 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.247962 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:55.248627 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.249803 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.251799 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.252588 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.252760 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.253476 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:55.254202 (+ 726us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:55.254362 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.254460 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.254539 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":881,"threads_started":1}
I20260620 23:05:55.257524 20383 tablet_replica.cc:439] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ded67fd82424dd6b1aa013e371cb694" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } } peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } }
I20260620 23:05:55.257826 20383 tablet_replica.cc:442] T 5c4adcc562b94ec2a6cd19d89d3dfafe P 741baf7e14d64762ba9449dcbb2e46d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:05:55.271783 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.265715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:36613 (local address 127.12.2.67:42755)
0620 23:05:55.266294 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:55.266310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:55.266423 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:55.266783 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:55.266787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:55.266796 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:55.267105 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.267113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.268593 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.268597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.269700 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.269714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.270709 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.270714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.270854 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.270907 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:55.271316 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:55.271324 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:05:55.271376 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 23:05:55.271472 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":309,"threads_started":1}
I20260620 23:05:55.271786 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.265802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:42755 (local address 127.12.2.66:36613)
0620 23:05:55.266006 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.266013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.266387 (+ 374us) server_negotiation.cc:407] Connection header received
0620 23:05:55.266566 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.266572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.266658 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.266754 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:55.267245 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.268472 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.269854 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.270542 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.270676 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.271099 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:55.271223 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:55.271320 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.271420 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.271497 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:05:55.281280 20027 raft_consensus.cc:1261] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Refusing update from remote peer 741baf7e14d64762ba9449dcbb2e46d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:55.281489 20166 raft_consensus.cc:1261] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Refusing update from remote peer 741baf7e14d64762ba9449dcbb2e46d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:55.281934 20407 consensus_queue.cc:1035] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:55.282214 20383 consensus_queue.cc:1035] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:55.316943 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.310261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:39831 (local address 127.12.2.67:45787)
0620 23:05:55.310429 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.310435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.311194 (+ 759us) server_negotiation.cc:407] Connection header received
0620 23:05:55.311400 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.311405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.311484 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.311574 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:55.312452 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.313530 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.315054 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.315693 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.315809 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.316291 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:55.316412 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:55.316494 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.316592 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.316666 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:05:55.316982 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.310268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:45787 (local address 127.12.2.66:39831)
0620 23:05:55.311093 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:55.311117 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:55.311253 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:55.311913 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:55.311917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:55.311929 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:55.312298 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.312309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.313644 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.313647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.314908 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.314922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.315840 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.315845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.316044 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.316111 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:55.316496 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:55.316505 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:05:55.316572 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:05:55.316673 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":341,"threads_started":1}
I20260620 23:05:55.354704 19866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:55.355000 19866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:55.358506 19866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5: Sending DeleteTablet for 3 replicas of tablet ab2d645b4a1e4ba0a5f0be87d73b0577
I20260620 23:05:55.359423 20141 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2d645b4a1e4ba0a5f0be87d73b0577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:55 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:33612
I20260620 23:05:55.359521 20003 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2d645b4a1e4ba0a5f0be87d73b0577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:55 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:35438
I20260620 23:05:55.367661 20281 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2d645b4a1e4ba0a5f0be87d73b0577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:55 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:51220
I20260620 23:05:55.368206 20443 tablet_replica.cc:330] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb: stopping tablet replica
I20260620 23:05:55.368417 20444 tablet_replica.cc:330] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694: stopping tablet replica
I20260620 23:05:55.368644 20443 raft_consensus.cc:2229] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:55.368960 20444 raft_consensus.cc:2229] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:55.369307 20443 raft_consensus.cc:2258] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:55.369603 20444 raft_consensus.cc:2258] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:55.370873 20444 ts_tablet_manager.cc:1905] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:55.371886 20443 ts_tablet_manager.cc:1905] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:55.383622 20445 tablet_replica.cc:330] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6: stopping tablet replica
I20260620 23:05:55.383875 20445 raft_consensus.cc:2229] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:55.384531 20445 raft_consensus.cc:2258] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:55.385583 20443 ts_tablet_manager.cc:1918] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:55.385896 20445 ts_tablet_manager.cc:1905] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:55.386898 20444 ts_tablet_manager.cc:1918] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:55.387187 20443 log.cc:1196] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/ab2d645b4a1e4ba0a5f0be87d73b0577
I20260620 23:05:55.387682 20443 ts_tablet_manager.cc:1939] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 1dd320521b0a45949654b6ad5163b6cb: Deleting consensus metadata
I20260620 23:05:55.387923 20444 log.cc:1196] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/ab2d645b4a1e4ba0a5f0be87d73b0577
I20260620 23:05:55.389099 19851 catalog_manager.cc:4841] TS 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65:43785): tablet ab2d645b4a1e4ba0a5f0be87d73b0577 (table test-table [id=d45c6be6355f43d58f6bc2980ca2ea66]) successfully deleted
I20260620 23:05:55.389784 20444 ts_tablet_manager.cc:1939] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 8ded67fd82424dd6b1aa013e371cb694: Deleting consensus metadata
I20260620 23:05:55.390899 19852 catalog_manager.cc:4841] TS 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66:36613): tablet ab2d645b4a1e4ba0a5f0be87d73b0577 (table test-table [id=d45c6be6355f43d58f6bc2980ca2ea66]) successfully deleted
I20260620 23:05:55.397224 20445 ts_tablet_manager.cc:1918] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:55.397377 20445 log.cc:1196] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/ab2d645b4a1e4ba0a5f0be87d73b0577
I20260620 23:05:55.397939 20445 ts_tablet_manager.cc:1939] T ab2d645b4a1e4ba0a5f0be87d73b0577 P 741baf7e14d64762ba9449dcbb2e46d6: Deleting consensus metadata
I20260620 23:05:55.399106 19850 catalog_manager.cc:4841] TS 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787): tablet ab2d645b4a1e4ba0a5f0be87d73b0577 (table test-table [id=d45c6be6355f43d58f6bc2980ca2ea66]) successfully deleted
Jun 20 23:05:55 dist-test-slave-7g8g krb5kdc[19821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:05:55 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:05:55 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:05:55.481087 20415 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:05:55.468173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.12.2.126:37269)
0620 23:05:55.468434 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.468441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.468972 (+ 531us) server_negotiation.cc:407] Connection header received
0620 23:05:55.469228 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.469234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.469313 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.469380 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:55.471246 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.472384 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.474235 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.474889 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.475021 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.480834 (+ 5813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41674: BlockingRecv error: recv got EOF from 127.0.0.1:41674 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Jun 20 23:05:55 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM
I20260620 23:05:55.516978 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.499643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.12.2.126:37269)
0620 23:05:55.499828 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.499834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.500307 (+ 473us) server_negotiation.cc:407] Connection header received
0620 23:05:55.500488 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.500493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.500577 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.500672 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:55.502221 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.503348 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.505366 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.506048 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.506172 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.510916 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:55.510923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:55.510957 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:55.510999 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:55.514174 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:55.514908 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:55.514914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:55.514918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:55.515107 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:55.515510 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:55.515515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:55.515518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:55.515859 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:55.516097 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.516390 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.516614 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:05:55.520691 19866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:55.521401 19866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:55.529013 20003 tablet_service.cc:1467] Processing CreateTablet for tablet 494e04a3b43745609ccd7ffabf491862 (DEFAULT_TABLE table=test-table [id=e771c3fef52d4248bad97b99be1ada9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:55.529313 20281 tablet_service.cc:1467] Processing CreateTablet for tablet 494e04a3b43745609ccd7ffabf491862 (DEFAULT_TABLE table=test-table [id=e771c3fef52d4248bad97b99be1ada9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:55.529449 20003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 494e04a3b43745609ccd7ffabf491862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:55.529747 20281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 494e04a3b43745609ccd7ffabf491862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:55.531692 20141 tablet_service.cc:1467] Processing CreateTablet for tablet 494e04a3b43745609ccd7ffabf491862 (DEFAULT_TABLE table=test-table [id=e771c3fef52d4248bad97b99be1ada9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:55.532128 20141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 494e04a3b43745609ccd7ffabf491862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:55.536607 20376 tablet_bootstrap.cc:492] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb: Bootstrap starting.
I20260620 23:05:55.538187 20376 tablet_bootstrap.cc:654] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:55.547000 20377 tablet_bootstrap.cc:492] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694: Bootstrap starting.
I20260620 23:05:55.548270 20376 tablet_bootstrap.cc:492] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb: No bootstrap required, opened a new log
I20260620 23:05:55.548518 20376 ts_tablet_manager.cc:1397] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.001s
I20260620 23:05:55.548789 20377 tablet_bootstrap.cc:654] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:55.549387 20376 raft_consensus.cc:348] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.549731 20376 raft_consensus.cc:374] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:55.549857 20376 raft_consensus.cc:729] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd320521b0a45949654b6ad5163b6cb, State: Initialized, Role: FOLLOWER
I20260620 23:05:55.550143 20376 consensus_queue.cc:260] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.552289 20376 ts_tablet_manager.cc:1428] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260620 23:05:55.554885 20375 tablet_bootstrap.cc:492] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6: Bootstrap starting.
I20260620 23:05:55.556397 20375 tablet_bootstrap.cc:654] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:55.556634 20377 tablet_bootstrap.cc:492] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694: No bootstrap required, opened a new log
I20260620 23:05:55.556790 20377 ts_tablet_manager.cc:1397] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260620 23:05:55.557561 20377 raft_consensus.cc:348] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.557844 20377 raft_consensus.cc:374] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:55.557947 20377 raft_consensus.cc:729] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ded67fd82424dd6b1aa013e371cb694, State: Initialized, Role: FOLLOWER
I20260620 23:05:55.558488 20375 tablet_bootstrap.cc:492] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6: No bootstrap required, opened a new log
I20260620 23:05:55.558737 20377 consensus_queue.cc:260] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.559315 20377 ts_tablet_manager.cc:1428] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:05:55.559691 20375 ts_tablet_manager.cc:1397] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 23:05:55.560469 20375 raft_consensus.cc:348] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.560801 20375 raft_consensus.cc:374] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:55.560937 20375 raft_consensus.cc:729] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741baf7e14d64762ba9449dcbb2e46d6, State: Initialized, Role: FOLLOWER
I20260620 23:05:55.561226 20375 consensus_queue.cc:260] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.561820 20375 ts_tablet_manager.cc:1428] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:55.689990 20381 raft_consensus.cc:482] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:55.690179 20381 raft_consensus.cc:504] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.691658 20381 leader_election.cc:288] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66:36613), 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787)
I20260620 23:05:55.698545 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.692483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:41087 (local address 127.12.2.66:36613)
0620 23:05:55.692690 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.692695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.692957 (+ 262us) server_negotiation.cc:407] Connection header received
0620 23:05:55.693106 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.693111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.693186 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.693286 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:55.693764 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.694837 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.696670 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.697720 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.697829 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.697934 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.698171 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.698300 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 23:05:55.698568 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.691936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:36613 (local address 127.12.2.65:41087)
0620 23:05:55.692857 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:55.692873 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:55.692989 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:55.693372 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:55.693377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:55.693399 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:55.693605 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.693617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.694998 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.695003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.696482 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.696497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.697858 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.697864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.698031 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.698055 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:05:55.698131 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:05:55.698267 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":776,"threads_started":1}
I20260620 23:05:55.699458 20165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "494e04a3b43745609ccd7ffabf491862" candidate_uuid: "1dd320521b0a45949654b6ad5163b6cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ded67fd82424dd6b1aa013e371cb694" is_pre_election: true
I20260620 23:05:55.699765 20165 raft_consensus.cc:2454] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd320521b0a45949654b6ad5163b6cb in term 0.
I20260620 23:05:55.699884 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.692206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:51393 (local address 127.12.2.67:45787)
0620 23:05:55.692378 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.692384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.693285 (+ 901us) server_negotiation.cc:407] Connection header received
0620 23:05:55.693434 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.693439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.693537 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.693600 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:55.694230 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.695006 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.697439 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.698886 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.699049 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.699151 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.699453 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.699595 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:05:55.700259 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.691950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:45787 (local address 127.12.2.65:51393)
0620 23:05:55.693180 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:55.693194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:55.693292 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:55.693780 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:55.693783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:55.693802 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:55.694060 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.694066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.695144 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.695153 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.697230 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.697245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.699139 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.699145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.699307 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.699331 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:05:55.699409 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:05:55.699529 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1154,"mutex_wait_us":1,"thread_start_us":516,"threads_started":1}
I20260620 23:05:55.700315 19955 leader_election.cc:302] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd320521b0a45949654b6ad5163b6cb, 8ded67fd82424dd6b1aa013e371cb694; no voters:
I20260620 23:05:55.700730 20381 raft_consensus.cc:2781] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:55.700850 20381 raft_consensus.cc:482] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:55.700928 20381 raft_consensus.cc:3037] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.700913 20301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "494e04a3b43745609ccd7ffabf491862" candidate_uuid: "1dd320521b0a45949654b6ad5163b6cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741baf7e14d64762ba9449dcbb2e46d6" is_pre_election: true
I20260620 23:05:55.701141 20301 raft_consensus.cc:2454] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd320521b0a45949654b6ad5163b6cb in term 0.
I20260620 23:05:55.703569 20381 raft_consensus.cc:504] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.704090 20381 leader_election.cc:288] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [CANDIDATE]: Term 1 election: Requested vote from peers 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66:36613), 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787)
I20260620 23:05:55.704556 20301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "494e04a3b43745609ccd7ffabf491862" candidate_uuid: "1dd320521b0a45949654b6ad5163b6cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741baf7e14d64762ba9449dcbb2e46d6"
I20260620 23:05:55.704586 20165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "494e04a3b43745609ccd7ffabf491862" candidate_uuid: "1dd320521b0a45949654b6ad5163b6cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ded67fd82424dd6b1aa013e371cb694"
I20260620 23:05:55.704775 20301 raft_consensus.cc:3037] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.704784 20165 raft_consensus.cc:3037] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.707703 20301 raft_consensus.cc:2454] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd320521b0a45949654b6ad5163b6cb in term 1.
I20260620 23:05:55.707703 20165 raft_consensus.cc:2454] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd320521b0a45949654b6ad5163b6cb in term 1.
I20260620 23:05:55.708061 19953 leader_election.cc:302] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd320521b0a45949654b6ad5163b6cb, 741baf7e14d64762ba9449dcbb2e46d6; no voters:
I20260620 23:05:55.708415 20381 raft_consensus.cc:2781] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:55.708834 20381 raft_consensus.cc:686] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 1 LEADER]: Becoming Leader. State: Replica: 1dd320521b0a45949654b6ad5163b6cb, State: Running, Role: LEADER
I20260620 23:05:55.709174 20381 consensus_queue.cc:237] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.712567 19866 catalog_manager.cc:5495] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65). New cstate: current_term: 1 leader_uuid: "1dd320521b0a45949654b6ad5163b6cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:55.750772 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.745287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.12.2.65:43785)
0620 23:05:55.745801 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.745806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.745824 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:05:55.745907 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.745911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.745984 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.746129 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:55.746546 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.747342 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.748611 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.749021 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.749126 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.749542 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:55.750122 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:55.750249 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.750343 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.750416 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":272,"threads_started":1}
I20260620 23:05:55.754808 20165 raft_consensus.cc:1261] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Refusing update from remote peer 1dd320521b0a45949654b6ad5163b6cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:55.754837 20301 raft_consensus.cc:1261] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Refusing update from remote peer 1dd320521b0a45949654b6ad5163b6cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:55.755610 20381 consensus_queue.cc:1035] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:55.755877 20463 consensus_queue.cc:1035] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:55.778888 19865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:55.779198 19865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:55.782688 19865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5: Sending DeleteTablet for 3 replicas of tablet 494e04a3b43745609ccd7ffabf491862
I20260620 23:05:55.784235 20281 tablet_service.cc:1514] Processing DeleteTablet for tablet 494e04a3b43745609ccd7ffabf491862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:55 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:51220
I20260620 23:05:55.784245 20003 tablet_service.cc:1514] Processing DeleteTablet for tablet 494e04a3b43745609ccd7ffabf491862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:55 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:35438
I20260620 23:05:55.784524 20443 tablet_replica.cc:330] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb: stopping tablet replica
I20260620 23:05:55.784605 20445 tablet_replica.cc:330] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6: stopping tablet replica
I20260620 23:05:55.784741 20443 raft_consensus.cc:2229] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:55.784825 20445 raft_consensus.cc:2229] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:55.785037 20445 raft_consensus.cc:2258] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:55.785743 20141 tablet_service.cc:1514] Processing DeleteTablet for tablet 494e04a3b43745609ccd7ffabf491862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:55 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:33612
I20260620 23:05:55.785996 20444 tablet_replica.cc:330] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694: stopping tablet replica
I20260620 23:05:55.786170 20444 raft_consensus.cc:2229] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:55.786326 20445 ts_tablet_manager.cc:1905] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:55.786432 20444 raft_consensus.cc:2258] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:55.786768 19865 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:55.787403 19865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:55.787585 20444 ts_tablet_manager.cc:1905] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:55.792606 20445 ts_tablet_manager.cc:1918] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:55.792760 20445 log.cc:1196] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/494e04a3b43745609ccd7ffabf491862
I20260620 23:05:55.793119 20445 ts_tablet_manager.cc:1939] T 494e04a3b43745609ccd7ffabf491862 P 741baf7e14d64762ba9449dcbb2e46d6: Deleting consensus metadata
I20260620 23:05:55.793596 20444 ts_tablet_manager.cc:1918] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:55.793731 20444 log.cc:1196] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/494e04a3b43745609ccd7ffabf491862
I20260620 23:05:55.794121 20444 ts_tablet_manager.cc:1939] T 494e04a3b43745609ccd7ffabf491862 P 8ded67fd82424dd6b1aa013e371cb694: Deleting consensus metadata
I20260620 23:05:55.794114 19850 catalog_manager.cc:4841] TS 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787): tablet 494e04a3b43745609ccd7ffabf491862 (table test-table [id=e771c3fef52d4248bad97b99be1ada9b]) successfully deleted
I20260620 23:05:55.794775 20281 tablet_service.cc:1467] Processing CreateTablet for tablet 160e4489afd446118948dfa4b080d629 (DEFAULT_TABLE table=test-table [id=3d7d593da7ea4716be3b8e45d8f2615e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:55.794967 20003 tablet_service.cc:1467] Processing CreateTablet for tablet 160e4489afd446118948dfa4b080d629 (DEFAULT_TABLE table=test-table [id=3d7d593da7ea4716be3b8e45d8f2615e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:55.795228 20281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160e4489afd446118948dfa4b080d629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:55.795365 20003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160e4489afd446118948dfa4b080d629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:55.799767 20141 tablet_service.cc:1467] Processing CreateTablet for tablet 160e4489afd446118948dfa4b080d629 (DEFAULT_TABLE table=test-table [id=3d7d593da7ea4716be3b8e45d8f2615e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:55.800137 20141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160e4489afd446118948dfa4b080d629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:55.802320 20376 tablet_bootstrap.cc:492] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb: Bootstrap starting.
I20260620 23:05:55.804132 20376 tablet_bootstrap.cc:654] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:55.807231 20377 tablet_bootstrap.cc:492] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694: Bootstrap starting.
I20260620 23:05:55.807720 20376 tablet_bootstrap.cc:492] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb: No bootstrap required, opened a new log
I20260620 23:05:55.808151 20376 ts_tablet_manager.cc:1397] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 23:05:55.808288 20375 tablet_bootstrap.cc:492] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6: Bootstrap starting.
I20260620 23:05:55.808830 20377 tablet_bootstrap.cc:654] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:55.809048 20376 raft_consensus.cc:348] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.809401 20376 raft_consensus.cc:374] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:55.809515 20376 raft_consensus.cc:729] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd320521b0a45949654b6ad5163b6cb, State: Initialized, Role: FOLLOWER
I20260620 23:05:55.809842 20375 tablet_bootstrap.cc:654] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:55.809799 20376 consensus_queue.cc:260] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.810377 20376 ts_tablet_manager.cc:1428] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:55.814148 19852 catalog_manager.cc:4841] TS 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66:36613): tablet 494e04a3b43745609ccd7ffabf491862 (table test-table [id=e771c3fef52d4248bad97b99be1ada9b]) successfully deleted
I20260620 23:05:55.815526 20443 raft_consensus.cc:2258] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:55.816871 20443 ts_tablet_manager.cc:1905] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:55.819603 20375 tablet_bootstrap.cc:492] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6: No bootstrap required, opened a new log
I20260620 23:05:55.819759 20375 ts_tablet_manager.cc:1397] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260620 23:05:55.820493 20375 raft_consensus.cc:348] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.820755 20375 raft_consensus.cc:374] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:55.820844 20375 raft_consensus.cc:729] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741baf7e14d64762ba9449dcbb2e46d6, State: Initialized, Role: FOLLOWER
I20260620 23:05:55.821115 20375 consensus_queue.cc:260] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.822503 20375 ts_tablet_manager.cc:1428] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 23:05:55.826762 20443 ts_tablet_manager.cc:1918] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:55.826922 20443 log.cc:1196] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/494e04a3b43745609ccd7ffabf491862
I20260620 23:05:55.827363 20443 ts_tablet_manager.cc:1939] T 494e04a3b43745609ccd7ffabf491862 P 1dd320521b0a45949654b6ad5163b6cb: Deleting consensus metadata
I20260620 23:05:55.831043 19851 catalog_manager.cc:4841] TS 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65:43785): tablet 494e04a3b43745609ccd7ffabf491862 (table test-table [id=e771c3fef52d4248bad97b99be1ada9b]) successfully deleted
I20260620 23:05:55.836465 20377 tablet_bootstrap.cc:492] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694: No bootstrap required, opened a new log
I20260620 23:05:55.836632 20377 ts_tablet_manager.cc:1397] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.009s
I20260620 23:05:55.837749 20377 raft_consensus.cc:348] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.837944 20377 raft_consensus.cc:374] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:55.838018 20377 raft_consensus.cc:729] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ded67fd82424dd6b1aa013e371cb694, State: Initialized, Role: FOLLOWER
I20260620 23:05:55.838243 20377 consensus_queue.cc:260] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.838794 20377 ts_tablet_manager.cc:1428] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 23:05:55.928378 20463 raft_consensus.cc:482] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:55.928629 20463 raft_consensus.cc:504] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.929421 20463 leader_election.cc:288] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66:36613), 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787)
I20260620 23:05:55.929920 20165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160e4489afd446118948dfa4b080d629" candidate_uuid: "1dd320521b0a45949654b6ad5163b6cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ded67fd82424dd6b1aa013e371cb694" is_pre_election: true
I20260620 23:05:55.930147 20165 raft_consensus.cc:2454] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd320521b0a45949654b6ad5163b6cb in term 0.
I20260620 23:05:55.930220 20301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160e4489afd446118948dfa4b080d629" candidate_uuid: "1dd320521b0a45949654b6ad5163b6cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741baf7e14d64762ba9449dcbb2e46d6" is_pre_election: true
I20260620 23:05:55.930516 20301 raft_consensus.cc:2454] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd320521b0a45949654b6ad5163b6cb in term 0.
I20260620 23:05:55.930665 19955 leader_election.cc:302] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd320521b0a45949654b6ad5163b6cb, 8ded67fd82424dd6b1aa013e371cb694; no voters:
I20260620 23:05:55.931053 20463 raft_consensus.cc:2781] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:55.931176 20463 raft_consensus.cc:482] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:55.931298 20463 raft_consensus.cc:3037] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.933894 20463 raft_consensus.cc:504] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.934866 20165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160e4489afd446118948dfa4b080d629" candidate_uuid: "1dd320521b0a45949654b6ad5163b6cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ded67fd82424dd6b1aa013e371cb694"
I20260620 23:05:55.934881 20301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160e4489afd446118948dfa4b080d629" candidate_uuid: "1dd320521b0a45949654b6ad5163b6cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741baf7e14d64762ba9449dcbb2e46d6"
I20260620 23:05:55.935089 20301 raft_consensus.cc:3037] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.935096 20165 raft_consensus.cc:3037] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.938076 20301 raft_consensus.cc:2454] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd320521b0a45949654b6ad5163b6cb in term 1.
I20260620 23:05:55.938086 20165 raft_consensus.cc:2454] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd320521b0a45949654b6ad5163b6cb in term 1.
I20260620 23:05:55.938541 19953 leader_election.cc:302] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd320521b0a45949654b6ad5163b6cb, 741baf7e14d64762ba9449dcbb2e46d6; no voters:
I20260620 23:05:55.938880 20469 raft_consensus.cc:2781] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:55.938973 20463 leader_election.cc:288] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [CANDIDATE]: Term 1 election: Requested vote from peers 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66:36613), 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787)
I20260620 23:05:55.939136 20469 raft_consensus.cc:686] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 1 LEADER]: Becoming Leader. State: Replica: 1dd320521b0a45949654b6ad5163b6cb, State: Running, Role: LEADER
I20260620 23:05:55.939512 20469 consensus_queue.cc:237] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } }
I20260620 23:05:55.943459 19866 catalog_manager.cc:5495] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65). New cstate: current_term: 1 leader_uuid: "1dd320521b0a45949654b6ad5163b6cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd320521b0a45949654b6ad5163b6cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:56.000141 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.994672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:60727 (local address 127.12.2.66:36613)
0620 23:05:55.994840 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.994846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.995434 (+ 588us) server_negotiation.cc:407] Connection header received
0620 23:05:55.995616 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.995620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.995696 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.995878 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:55.996400 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.997114 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.998528 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.998964 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.999070 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.999468 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:55.999564 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:55.999635 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.999770 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.999844 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:05:56.000240 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.994669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:36613 (local address 127.12.2.65:60727)
0620 23:05:55.995340 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:55.995356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:55.995494 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:55.995858 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:55.995863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:55.995877 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:55.996238 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.996249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.997240 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.997245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.998401 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:55.998415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.999127 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.999134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:55.999265 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.999317 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:55.999649 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:55.999660 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:05:55.999739 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:05:55.999870 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":346,"threads_started":1}
I20260620 23:05:56.009159 20301 raft_consensus.cc:1261] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Refusing update from remote peer 1dd320521b0a45949654b6ad5163b6cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:56.009212 20165 raft_consensus.cc:1261] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Refusing update from remote peer 1dd320521b0a45949654b6ad5163b6cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:56.009786 20381 consensus_queue.cc:1035] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ded67fd82424dd6b1aa013e371cb694" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:56.010078 20469 consensus_queue.cc:1035] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "741baf7e14d64762ba9449dcbb2e46d6" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:56.048560 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:56.031730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:43785 (local address 127.12.2.66:47393)
0620 23:05:56.043352 (+ 11622us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:56.043369 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:56.043478 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:56.043781 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:56.043784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:56.043794 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:05:56.043963 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:56.043969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.045408 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.045414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:56.046707 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:56.046719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.047626 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.047630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:56.047755 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:56.047794 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:05:56.048136 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:05:56.048142 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:05:56.048192 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 23:05:56.048302 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11479,"thread_start_us":11368,"threads_started":1}
I20260620 23:05:56.048609 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:56.031888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:47393 (local address 127.12.2.65:43785)
0620 23:05:56.032046 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:56.032052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:56.043462 (+ 11410us) server_negotiation.cc:407] Connection header received
0620 23:05:56.043581 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:56.043591 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:56.043681 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:56.043780 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:56.044094 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.045267 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.046836 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.047507 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.047630 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:56.047934 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:56.048060 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:56.048141 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:56.048220 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 23:05:56.048306 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:05:56.081331 19865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:56.081629 19865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:56.084857 19865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a9df4e2ca774683b1f0014599bcc7e5: Sending DeleteTablet for 3 replicas of tablet 160e4489afd446118948dfa4b080d629
I20260620 23:05:56.085702 20003 tablet_service.cc:1514] Processing DeleteTablet for tablet 160e4489afd446118948dfa4b080d629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:56 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:35438
I20260620 23:05:56.085729 20141 tablet_service.cc:1514] Processing DeleteTablet for tablet 160e4489afd446118948dfa4b080d629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:56 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:33612
I20260620 23:05:56.085949 20281 tablet_service.cc:1514] Processing DeleteTablet for tablet 160e4489afd446118948dfa4b080d629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:56 UTC) from {username='oryx', principal='oryx/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:51220
I20260620 23:05:56.086015 20444 tablet_replica.cc:330] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694: stopping tablet replica
I20260620 23:05:56.086011 20443 tablet_replica.cc:330] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb: stopping tablet replica
I20260620 23:05:56.086177 20444 raft_consensus.cc:2229] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:56.086184 20443 raft_consensus.cc:2229] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:56.086226 20445 tablet_replica.cc:330] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6: stopping tablet replica
I20260620 23:05:56.086364 20444 raft_consensus.cc:2258] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:56.086400 20445 raft_consensus.cc:2229] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:56.086519 20443 raft_consensus.cc:2258] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:56.086588 20445 raft_consensus.cc:2258] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:56.087483 20443 ts_tablet_manager.cc:1905] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:56.087721 20445 ts_tablet_manager.cc:1905] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:56.093255 20443 ts_tablet_manager.cc:1918] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:56.093391 20443 log.cc:1196] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/160e4489afd446118948dfa4b080d629
I20260620 23:05:56.093758 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 19937
I20260620 23:05:56.093842 20443 ts_tablet_manager.cc:1939] T 160e4489afd446118948dfa4b080d629 P 1dd320521b0a45949654b6ad5163b6cb: Deleting consensus metadata
I20260620 23:05:56.093827 20445 ts_tablet_manager.cc:1918] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:56.093946 20445 log.cc:1196] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/160e4489afd446118948dfa4b080d629
I20260620 23:05:56.094352 20445 ts_tablet_manager.cc:1939] T 160e4489afd446118948dfa4b080d629 P 741baf7e14d64762ba9449dcbb2e46d6: Deleting consensus metadata
I20260620 23:05:56.095273 19851 catalog_manager.cc:4841] TS 1dd320521b0a45949654b6ad5163b6cb (127.12.2.65:43785): tablet 160e4489afd446118948dfa4b080d629 (table test-table [id=3d7d593da7ea4716be3b8e45d8f2615e]) successfully deleted
I20260620 23:05:56.095376 19850 catalog_manager.cc:4841] TS 741baf7e14d64762ba9449dcbb2e46d6 (127.12.2.67:45787): tablet 160e4489afd446118948dfa4b080d629 (table test-table [id=3d7d593da7ea4716be3b8e45d8f2615e]) successfully deleted
I20260620 23:05:56.095521 20444 ts_tablet_manager.cc:1905] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:56.101536 20444 ts_tablet_manager.cc:1918] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:05:56.101677 20444 log.cc:1196] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/160e4489afd446118948dfa4b080d629
I20260620 23:05:56.102100 20444 ts_tablet_manager.cc:1939] T 160e4489afd446118948dfa4b080d629 P 8ded67fd82424dd6b1aa013e371cb694: Deleting consensus metadata
I20260620 23:05:56.103144 19852 catalog_manager.cc:4841] TS 8ded67fd82424dd6b1aa013e371cb694 (127.12.2.66:36613): tablet 160e4489afd446118948dfa4b080d629 (table test-table [id=3d7d593da7ea4716be3b8e45d8f2615e]) successfully deleted
Jun 20 23:05:56 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.2.65@KRBTEST.COM
I20260620 23:05:56.120052 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:56.099563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.12.2.65:43785)
0620 23:05:56.099749 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:56.099756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:56.099780 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:05:56.099837 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:56.099842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:56.099938 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:56.100049 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:56.101594 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.102687 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.104526 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.104997 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.105097 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:56.114356 (+ 9259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:56.114368 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:56.114383 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:56.114425 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:56.117339 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:56.118125 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:56.118131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:56.118133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:56.118260 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:56.118638 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:56.118641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:56.118643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:56.119028 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:56.119493 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:56.119608 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 23:05:56.119764 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:05:56.120779 20068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:56.263332 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 19937
I20260620 23:05:56.280839 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 20077
Jun 20 23:05:56 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.2.66@KRBTEST.COM
I20260620 23:05:56.294268 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:56.281673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.12.2.66:36613)
0620 23:05:56.281860 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:56.281864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:56.281880 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:05:56.281935 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:56.281939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:56.282005 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:56.282083 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:56.283232 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.283900 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.285070 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.285483 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.285571 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:56.289335 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:56.289349 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:56.289365 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:56.289395 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:56.291964 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:56.292560 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:56.292567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:56.292569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:56.292688 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:56.292982 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:56.292985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:56.292987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:56.293370 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:56.293585 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:56.293810 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 23:05:56.293967 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:05:56.294850 20207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:56.439564 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 20077
I20260620 23:05:56.463286 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 20216
Jun 20 23:05:56 dist-test-slave-7g8g krb5kdc[19821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.2.67@KRBTEST.COM
I20260620 23:05:56.477378 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:56.463971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.12.2.67:45787)
0620 23:05:56.464427 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:56.464431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:56.464445 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:05:56.464493 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:56.464497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:56.464556 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:56.464645 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:56.465912 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.466636 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.467910 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.468296 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.468401 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:56.472456 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:56.472471 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:56.472487 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:56.472525 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:56.475000 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:56.475636 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:56.475643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:56.475646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:56.475843 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:56.476142 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:56.476146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:56.476148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:56.476504 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:56.476744 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:56.476947 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 23:05:56.477107 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":254,"threads_started":1}
I20260620 23:05:56.478062 20346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:56.596263 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 20216
I20260620 23:05:56.613289 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 19834
I20260620 23:05:56.614437 19906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:05:56.726121 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 19834
2026-06-20T23:05:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5469 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:56 dist-test-slave-7g8g krb5kdc[20493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:56 dist-test-slave-7g8g krb5kdc[20493](info): set up 2 sockets
Jun 20 23:05:56 dist-test-slave-7g8g krb5kdc[20493](info): commencing operation
krb5kdc: starting...
W20260620 23:05:57.723631 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:57 dist-test-slave-7g8g krb5kdc[20493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:05:57.926357 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:39063
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:58.237404 20506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:58.237697 20506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:58.237828 20506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:58.245934 20506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:58.246040 20506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:58.246105 20506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:58.246181 20506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:58.246243 20506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:58.256309 20506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:39063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:58.257644 20506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:58.258991 20506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:58.270799 20512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:58.270880 20511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:58.271293 20506 server_base.cc:1034] running on GCE node
W20260620 23:05:58.270838 20514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:58.272095 20506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:58.273406 20506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:58.274610 20506 hybrid_clock.cc:648] HybridClock initialized: now 1781996758274556 us; error 37 us; skew 500 ppm
I20260620 23:05:58.274914 20506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T23:05:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1610 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:58 dist-test-slave-7g8g krb5kdc[20520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:58 dist-test-slave-7g8g krb5kdc[20520](info): set up 2 sockets
Jun 20 23:05:58 dist-test-slave-7g8g krb5kdc[20520](info): commencing operation
krb5kdc: starting...
W20260620 23:05:59.347859 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:05:59 dist-test-slave-7g8g krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:05:59.550572 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:34119
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:34119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:59.853986 20533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:59.854261 20533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:59.854369 20533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:59.861500 20533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:59.861614 20533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:59.861671 20533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:59.861757 20533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:59.861810 20533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:59.871788 20533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:34119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:34119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:05:59.873067 20533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:59.874323 20533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:59.887344 20538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:59.887344 20541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:59.887373 20539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:59.888459 20533 server_base.cc:1034] running on GCE node
I20260620 23:05:59.889238 20533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:59.890606 20533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:59.891788 20533 hybrid_clock.cc:648] HybridClock initialized: now 1781996759891737 us; error 45 us; skew 500 ppm
I20260620 23:05:59.892091 20533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:05:59 dist-test-slave-7g8g krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996759, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:59.894881 20533 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:05:59.896267 20533 webserver.cc:460] Webserver started at http://127.12.2.126:39419/ using document root <none> and password file <none>
I20260620 23:05:59.896910 20533 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:59.897017 20533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:59.897284 20533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:59.900066 20533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "22898450805144a5a1673207360c9110"
format_stamp: "Formatted at 2026-06-20 23:05:59 on dist-test-slave-7g8g"
server_key: "944ce405ef122ffec2c9d42c473f35ed"
server_key_iv: "e31fdf2085e235af7b7837fb1f558b71"
server_key_version: "encryptionkey@0"
I20260620 23:05:59.900754 20533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "22898450805144a5a1673207360c9110"
format_stamp: "Formatted at 2026-06-20 23:05:59 on dist-test-slave-7g8g"
server_key: "944ce405ef122ffec2c9d42c473f35ed"
server_key_iv: "e31fdf2085e235af7b7837fb1f558b71"
server_key_version: "encryptionkey@0"
I20260620 23:05:59.905615 20533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:05:59.909065 20548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:59.910377 20533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:59.910528 20533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "22898450805144a5a1673207360c9110"
format_stamp: "Formatted at 2026-06-20 23:05:59 on dist-test-slave-7g8g"
server_key: "944ce405ef122ffec2c9d42c473f35ed"
server_key_iv: "e31fdf2085e235af7b7837fb1f558b71"
server_key_version: "encryptionkey@0"
I20260620 23:05:59.910696 20533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:59.923308 20533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:59.927481 20533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:59.927775 20533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:59.945549 20533 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:34119
I20260620 23:05:59.945580 20599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:34119 every 8 connection(s)
I20260620 23:05:59.947018 20533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:59.949368 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 20533
I20260620 23:05:59.949569 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:05:59.950358 12297 external_mini_cluster.cc:1428] Setting key be66ce2fc53805d4e8e3fe066d151fc7
I20260620 23:05:59.951141 20600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:59.964632 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110: Bootstrap starting.
Jun 20 23:05:59 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:05:59.967761 20600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:59.968801 20600 log.cc:824] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:59.971410 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110: No bootstrap required, opened a new log
I20260620 23:05:59.972884 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:59.952415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.12.2.126:34119)
0620 23:05:59.953209 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:59.953220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:59.953261 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:05:59.955573 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:59.955594 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:59.955986 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:59.956318 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:59.958065 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:59.959421 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:59.961122 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:59.961884 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:59.962018 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:59.966302 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:59.966310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:59.966323 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:59.966371 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:59.969280 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:59.969922 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:59.969928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:59.969933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:59.970086 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:59.970427 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:59.970430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:59.970432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:59.970904 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:59.971180 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:59.971575 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 23:05:59.971954 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":346,"threads_started":1}
I20260620 23:05:59.979072 20600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } }
I20260620 23:05:59.979429 20600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:59.979518 20600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22898450805144a5a1673207360c9110, State: Initialized, Role: FOLLOWER
I20260620 23:05:59.980013 20600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } }
I20260620 23:05:59.980211 20600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:59.980329 20600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:59.980472 20600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:59.982641 20600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } }
I20260620 23:05:59.983161 20600 leader_election.cc:302] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22898450805144a5a1673207360c9110; no voters:
I20260620 23:05:59.983686 20600 leader_election.cc:288] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:59.983815 20605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:59.984450 20605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 1 LEADER]: Becoming Leader. State: Replica: 22898450805144a5a1673207360c9110, State: Running, Role: LEADER
I20260620 23:05:59.984952 20605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } }
I20260620 23:05:59.985589 20600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:59.987565 20607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22898450805144a5a1673207360c9110. Latest consensus state: current_term: 1 leader_uuid: "22898450805144a5a1673207360c9110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } } }
I20260620 23:05:59.987932 20607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:59.988850 20606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22898450805144a5a1673207360c9110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } } }
I20260620 23:05:59.989212 20606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:59.989531 20612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:59.995687 20612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:06:00.004979 20612 catalog_manager.cc:1347] Generated new cluster ID: eceab26f75cd46de8525b81d43510161
I20260620 23:06:00.005098 20612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:06:00.059402 20612 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:06:00.060688 20612 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:06:00.073436 20612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110: Generated new TSK 0
I20260620 23:06:00.074288 20612 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:06:00.130694 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34119
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:00.439494 20626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:00.439755 20626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:00.439882 20626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:00.447027 20626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:00.447135 20626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:00.447299 20626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:06:00.456774 20626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:00.458035 20626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:00.459329 20626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:00.471670 20631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:00.471642 20634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:00.471609 20632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:00.472823 20626 server_base.cc:1034] running on GCE node
I20260620 23:06:00.473585 20626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:00.474478 20626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:00.475664 20626 hybrid_clock.cc:648] HybridClock initialized: now 1781996760475623 us; error 38 us; skew 500 ppm
I20260620 23:06:00.475973 20626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:06:00 dist-test-slave-7g8g krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996760, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:00.478890 20626 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:06:00.480310 20626 webserver.cc:460] Webserver started at http://127.12.2.65:43437/ using document root <none> and password file <none>
I20260620 23:06:00.481034 20626 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:00.481153 20626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:00.481405 20626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:00.484159 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "1afcac0e51c34a31b308ca41528d6fc4"
format_stamp: "Formatted at 2026-06-20 23:06:00 on dist-test-slave-7g8g"
server_key: "82267c7cccd32637da02b48a0ace74d8"
server_key_iv: "29d77b2d488e85354b77fed1986f7828"
server_key_version: "encryptionkey@0"
I20260620 23:06:00.484838 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "1afcac0e51c34a31b308ca41528d6fc4"
format_stamp: "Formatted at 2026-06-20 23:06:00 on dist-test-slave-7g8g"
server_key: "82267c7cccd32637da02b48a0ace74d8"
server_key_iv: "29d77b2d488e85354b77fed1986f7828"
server_key_version: "encryptionkey@0"
I20260620 23:06:00.489792 20626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:06:00.493187 20641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:00.494534 20626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:06:00.494705 20626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "1afcac0e51c34a31b308ca41528d6fc4"
format_stamp: "Formatted at 2026-06-20 23:06:00 on dist-test-slave-7g8g"
server_key: "82267c7cccd32637da02b48a0ace74d8"
server_key_iv: "29d77b2d488e85354b77fed1986f7828"
server_key_version: "encryptionkey@0"
I20260620 23:06:00.494978 20626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:00.514518 20626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:00.518786 20626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:00.519085 20626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:00.520037 20626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:00.521349 20626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:00.521426 20626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:00.521543 20626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:00.521595 20626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:00.552841 20626 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:32881
I20260620 23:06:00.552881 20753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:32881 every 8 connection(s)
I20260620 23:06:00.554150 20626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:06:00.559780 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 20626
I20260620 23:06:00.559970 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:06:00.560571 12297 external_mini_cluster.cc:1428] Setting key a80c5656e6f90c1df0289ea020e45ef2
Jun 20 23:06:00 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996760, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:06:00.573784 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:00.556622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:59379 (local address 127.12.2.126:34119)
0620 23:06:00.557088 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:00.557093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:00.558097 (+ 1004us) server_negotiation.cc:407] Connection header received
0620 23:06:00.559980 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:00.559985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:00.560072 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:00.560167 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:00.561725 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:00.562470 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:00.564040 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:00.564424 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:00.564531 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:00.567773 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:00.567777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:00.567781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:00.567812 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:00.570456 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:00.571265 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:00.571271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:00.571273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:00.571399 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:00.571834 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:00.571837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:00.571839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:00.572160 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:00.572300 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:00.573087 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 23:06:00.573316 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":248,"threads_started":1}
I20260620 23:06:00.574777 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:00.557210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:59379)
0620 23:06:00.557937 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:00.557978 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:00.559701 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:00.560436 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:00.560445 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:00.560974 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:00.561551 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:00.561568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:00.562614 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:00.562618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:00.563908 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:00.563920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:00.564635 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:00.564642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:00.564864 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:00.565470 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:00.565498 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:00.567556 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:00.570610 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:00.570621 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:00.570639 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:00.571137 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:00.571508 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:00.571514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:00.571518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:00.571712 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:00.572289 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:00.572298 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:00.572678 (+ 380us) client_negotiation.cc:769] Sending connection context
0620 23:06:00.572948 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 23:06:00.573535 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":278,"threads_started":1}
I20260620 23:06:00.576546 20754 heartbeater.cc:347] Connected to a master server at 127.12.2.126:34119
I20260620 23:06:00.576874 20754 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:00.577651 20754 heartbeater.cc:511] Master 127.12.2.126:34119 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
I20260620 23:06:00.579862 20565 ts_manager.cc:194] Registered new tserver with Master: 1afcac0e51c34a31b308ca41528d6fc4 (127.12.2.65:32881)
I20260620 23:06:00.581456 20565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:59379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:06:00.605268 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34119
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:00.914671 20760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:00.914978 20760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:00.915082 20760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:00.922163 20760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:00.922271 20760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:00.922381 20760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:06:00.931887 20760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:00.933100 20760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:00.934345 20760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:00.947687 20765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:00.947666 20768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:00.947732 20766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:00.949184 20760 server_base.cc:1034] running on GCE node
I20260620 23:06:00.949863 20760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:00.950973 20760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:00.952201 20760 hybrid_clock.cc:648] HybridClock initialized: now 1781996760952149 us; error 42 us; skew 500 ppm
I20260620 23:06:00.952555 20760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:06:00 dist-test-slave-7g8g krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996760, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:00.956010 20760 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:06:00.962455 20760 webserver.cc:460] Webserver started at http://127.12.2.66:36211/ using document root <none> and password file <none>
I20260620 23:06:00.963178 20760 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:00.963289 20760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:00.963728 20760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:00.966881 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "e7359bb69ef04494bc2527555b6eeadf"
format_stamp: "Formatted at 2026-06-20 23:06:00 on dist-test-slave-7g8g"
server_key: "9d13e7ea183e80851fb6a0400cf7b453"
server_key_iv: "f1ab08cf3cbb04e3e123e6f04134b746"
server_key_version: "encryptionkey@0"
I20260620 23:06:00.967653 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "e7359bb69ef04494bc2527555b6eeadf"
format_stamp: "Formatted at 2026-06-20 23:06:00 on dist-test-slave-7g8g"
server_key: "9d13e7ea183e80851fb6a0400cf7b453"
server_key_iv: "f1ab08cf3cbb04e3e123e6f04134b746"
server_key_version: "encryptionkey@0"
I20260620 23:06:00.972554 20760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:06:00.976073 20775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:00.977473 20760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:06:00.977630 20760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "e7359bb69ef04494bc2527555b6eeadf"
format_stamp: "Formatted at 2026-06-20 23:06:00 on dist-test-slave-7g8g"
server_key: "9d13e7ea183e80851fb6a0400cf7b453"
server_key_iv: "f1ab08cf3cbb04e3e123e6f04134b746"
server_key_version: "encryptionkey@0"
I20260620 23:06:00.977854 20760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:00.993892 20760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:00.998193 20760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:00.998464 20760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:00.999497 20760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:01.000818 20760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:01.000892 20760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:01.001016 20760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:01.001066 20760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:01.030748 20760 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:41065
I20260620 23:06:01.030817 20887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:41065 every 8 connection(s)
I20260620 23:06:01.032032 20760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:06:01.035246 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 20760
I20260620 23:06:01.035400 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:06:01.035954 12297 external_mini_cluster.cc:1428] Setting key b739cdc03214aaaf359c8a6a26dd9e79
Jun 20 23:06:01 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996760, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:06:01.053427 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:01.034509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:33641 (local address 127.12.2.126:34119)
0620 23:06:01.034701 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:01.034707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:01.036258 (+ 1551us) server_negotiation.cc:407] Connection header received
0620 23:06:01.038127 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:01.038131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:01.038207 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:01.038283 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:01.040299 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.041019 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.043070 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.043459 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.043548 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:01.047446 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:01.047452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:01.047456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:01.047495 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:01.050415 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:01.051182 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:01.051187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:01.051191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:01.051344 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:01.051718 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:01.051722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:01.051725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:01.052044 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:01.052172 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:01.052896 (+ 724us) server_negotiation.cc:299] Negotiation successful
0620 23:06:01.053117 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:06:01.054318 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:01.035114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:33641)
0620 23:06:01.036075 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:01.036128 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:01.037848 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:01.038553 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:01.038565 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:01.039320 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:01.040113 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:01.040138 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.041163 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.041168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:01.042953 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:01.042968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.043661 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.043667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:01.043966 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:01.044645 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:01.044672 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:01.047195 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:01.050549 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:01.050556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:01.050570 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:01.051071 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:01.051435 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:01.051439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:01.051441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:01.051622 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:01.052165 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:01.052171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:01.052515 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 23:06:01.052777 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 23:06:01.053179 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":388,"threads_started":1}
I20260620 23:06:01.055763 20888 heartbeater.cc:347] Connected to a master server at 127.12.2.126:34119
I20260620 23:06:01.056118 20888 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:01.057017 20888 heartbeater.cc:511] Master 127.12.2.126:34119 requested a full tablet report, sending...
I20260620 23:06:01.058687 20565 ts_manager.cc:194] Registered new tserver with Master: e7359bb69ef04494bc2527555b6eeadf (127.12.2.66:41065)
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
I20260620 23:06:01.059551 20565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:33641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:06:01.086964 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34119
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:01.406319 20893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:01.406582 20893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:01.406723 20893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:01.413806 20893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:01.413933 20893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:01.414088 20893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:06:01.423651 20893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:01.424856 20893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:01.426101 20893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:01.438205 20898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:01.438233 20899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:01.438292 20901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:01.438881 20893 server_base.cc:1034] running on GCE node
I20260620 23:06:01.439453 20893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:01.440294 20893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:01.441473 20893 hybrid_clock.cc:648] HybridClock initialized: now 1781996761441421 us; error 50 us; skew 500 ppm
I20260620 23:06:01.441759 20893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:06:01 dist-test-slave-7g8g krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996761, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:01.447863 20893 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:06:01.449294 20893 webserver.cc:460] Webserver started at http://127.12.2.67:41995/ using document root <none> and password file <none>
I20260620 23:06:01.449985 20893 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:01.450088 20893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:01.450371 20893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:01.453191 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "3004aabf838342df95c604b07d4627d8"
format_stamp: "Formatted at 2026-06-20 23:06:01 on dist-test-slave-7g8g"
server_key: "a1c22baadbae438bfc8426d53e1edaee"
server_key_iv: "3c55c933461cb4b9e4e131affcc0241c"
server_key_version: "encryptionkey@0"
I20260620 23:06:01.453917 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "3004aabf838342df95c604b07d4627d8"
format_stamp: "Formatted at 2026-06-20 23:06:01 on dist-test-slave-7g8g"
server_key: "a1c22baadbae438bfc8426d53e1edaee"
server_key_iv: "3c55c933461cb4b9e4e131affcc0241c"
server_key_version: "encryptionkey@0"
I20260620 23:06:01.458998 20893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:06:01.462275 20908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:01.463603 20893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:06:01.463745 20893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "3004aabf838342df95c604b07d4627d8"
format_stamp: "Formatted at 2026-06-20 23:06:01 on dist-test-slave-7g8g"
server_key: "a1c22baadbae438bfc8426d53e1edaee"
server_key_iv: "3c55c933461cb4b9e4e131affcc0241c"
server_key_version: "encryptionkey@0"
I20260620 23:06:01.463923 20893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:01.486076 20893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:01.489776 20893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:01.490026 20893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:01.490947 20893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:01.492206 20893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:01.492280 20893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:01.492390 20893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:01.492439 20893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:01.521071 20893 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:39847
I20260620 23:06:01.521138 21020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:39847 every 8 connection(s)
I20260620 23:06:01.522251 20893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:06:01.527325 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 20893
I20260620 23:06:01.527499 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:06:01.528023 12297 external_mini_cluster.cc:1428] Setting key 8be80180f18469a1d6ae0cff1434f0c4
Jun 20 23:06:01 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996761, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:06:01.540820 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:01.524600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:43523 (local address 127.12.2.126:34119)
0620 23:06:01.524761 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:01.524765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:01.525895 (+ 1130us) server_negotiation.cc:407] Connection header received
0620 23:06:01.527992 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:01.527997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:01.528069 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:01.528179 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:01.529983 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.530782 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.532395 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.532793 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.532888 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:01.536060 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:01.536065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:01.536068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:01.536100 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:01.538274 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:01.538922 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:01.538925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:01.538942 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:01.539055 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:01.539436 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:01.539439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:01.539441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:01.539679 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:01.539797 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:01.540426 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 23:06:01.540586 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:06:01.541894 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:01.525084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:43523)
0620 23:06:01.525745 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:01.525780 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:01.527649 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:01.528483 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:01.528494 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:01.529022 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:01.529792 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:01.529823 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.530958 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.530964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:01.532258 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:01.532271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.532960 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.532964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:01.533159 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:01.533751 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:01.533771 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:01.535877 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:01.538406 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:01.538413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:01.538425 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:01.538832 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:01.539152 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:01.539156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:01.539159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:01.539351 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:01.539807 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:01.539813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:01.540123 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 23:06:01.540360 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 23:06:01.540752 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":254,"threads_started":1}
I20260620 23:06:01.543222 21021 heartbeater.cc:347] Connected to a master server at 127.12.2.126:34119
I20260620 23:06:01.543501 21021 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:01.544154 21021 heartbeater.cc:511] Master 127.12.2.126:34119 requested a full tablet report, sending...
I20260620 23:06:01.545459 20565 ts_manager.cc:194] Registered new tserver with Master: 3004aabf838342df95c604b07d4627d8 (127.12.2.67:39847)
I20260620 23:06:01.546274 20565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:43523
I20260620 23:06:01.555552 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.65@KRBTEST.COM" created.
I20260620 23:06:01.584625 20754 heartbeater.cc:503] Master 127.12.2.126:34119 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab.
Entry for principal oryx/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab.
WARNING: no policy specified for oryx/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab.
Entry for principal oryx/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab.
WARNING: no policy specified for oryx/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab.
Entry for principal oryx/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab.
I20260620 23:06:01.677562 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 20626
Jun 20 23:06:01 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:06:01.693308 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:01.678644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.12.2.65:32881)
0620 23:06:01.679252 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:01.679259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:01.679288 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:06:01.679357 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:01.679363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:01.679590 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:01.679724 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:01.681009 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.682067 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.683434 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.683985 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.684079 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:01.687936 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:01.687949 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:01.687964 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:01.687998 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:01.690889 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:01.691544 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:01.691551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:01.691554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:01.691690 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:01.691990 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:01.691993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:01.691995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:01.692340 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:01.692566 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:01.692778 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 23:06:01.692972 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":318,"threads_started":1}
I20260620 23:06:01.694572 20749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:01.776114 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 20626
I20260620 23:06:01.789451 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 20760
Jun 20 23:06:01 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:06:01.804620 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:01.790263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.12.2.66:41065)
0620 23:06:01.790750 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:01.790757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:01.790815 (+ 58us) server_negotiation.cc:407] Connection header received
0620 23:06:01.790875 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:01.790881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:01.791136 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:01.791283 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:01.792391 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.793364 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.794641 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.795320 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.795420 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:01.799573 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:01.799585 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:01.799601 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:01.799636 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:01.802337 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:01.802964 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:01.802973 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:01.802976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:01.803092 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:01.803368 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:01.803370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:01.803372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:01.803678 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:01.803902 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:01.804166 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 23:06:01.804350 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":224,"threads_started":1}
I20260620 23:06:01.805850 20883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:01.885192 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 20760
I20260620 23:06:01.897507 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 20893
Jun 20 23:06:01 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:06:01.913393 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:01.898394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44398 (local address 127.12.2.67:39847)
0620 23:06:01.898998 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:01.899005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:01.899034 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:06:01.899089 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:01.899095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:01.899317 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:01.899454 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:01.900780 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.901802 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.903108 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:01.903674 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:01.903778 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:01.907946 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:01.907959 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:01.907975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:01.908014 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:01.910673 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:01.911488 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:01.911495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:01.911497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:01.911644 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:01.912031 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:01.912034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:01.912036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:01.912392 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:01.912630 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:01.912887 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 23:06:01.913091 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":290,"threads_started":1}
I20260620 23:06:01.914906 21016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:01.996975 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 20893
I20260620 23:06:02.009371 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 20533
I20260620 23:06:02.010555 20595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:02.109426 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 20533
I20260620 23:06:02.125696 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:34119
--webserver_interface=127.12.2.126
--webserver_port=39419
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:34119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:02.436913 21038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:02.437189 21038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:02.437275 21038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:02.444329 21038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:02.444434 21038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:02.444478 21038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:02.444533 21038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:02.444576 21038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:02.454248 21038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:34119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:34119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=39419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:02.455545 21038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:02.456763 21038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:02.467556 21044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:02.467504 21046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:02.467913 21043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:02.468291 21038 server_base.cc:1034] running on GCE node
I20260620 23:06:02.469010 21038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:02.470278 21038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:02.471482 21038 hybrid_clock.cc:648] HybridClock initialized: now 1781996762471424 us; error 40 us; skew 500 ppm
I20260620 23:06:02.471789 21038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:06:02 dist-test-slave-7g8g krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996762, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:02.474653 21038 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:06:02.476274 21038 webserver.cc:460] Webserver started at http://127.12.2.126:39419/ using document root <none> and password file <none>
I20260620 23:06:02.477054 21038 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:02.477187 21038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:02.483038 21038 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:06:02.485635 21053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:02.487006 21038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:06:02.487150 21038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "22898450805144a5a1673207360c9110"
format_stamp: "Formatted at 2026-06-20 23:05:59 on dist-test-slave-7g8g"
server_key: "944ce405ef122ffec2c9d42c473f35ed"
server_key_iv: "e31fdf2085e235af7b7837fb1f558b71"
server_key_version: "encryptionkey@0"
I20260620 23:06:02.488070 21038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:02.507285 21038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:02.511250 21038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:02.511474 21038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:02.528007 21038 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:34119
I20260620 23:06:02.528065 21104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:34119 every 8 connection(s)
I20260620 23:06:02.529414 21038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:06:02.535025 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 21038
I20260620 23:06:02.535760 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:32881
--local_ip_for_outbound_sockets=127.12.2.65
--tserver_master_addrs=127.12.2.126:34119
--webserver_port=43437
--webserver_interface=127.12.2.65
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.2.65
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:06:02.536396 21105 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:06:02.539757 21105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110: Bootstrap starting.
I20260620 23:06:02.561699 21105 log.cc:824] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:02.570544 21105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:06:02.571116 21105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110: Bootstrap complete.
I20260620 23:06:02.579736 21105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } }
I20260620 23:06:02.580542 21105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22898450805144a5a1673207360c9110, State: Initialized, Role: FOLLOWER
I20260620 23:06:02.581169 21105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } }
I20260620 23:06:02.581382 21105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:06:02.581487 21105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:06:02.581630 21105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:06:02.585224 21105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } }
I20260620 23:06:02.585719 21105 leader_election.cc:302] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22898450805144a5a1673207360c9110; no voters:
I20260620 23:06:02.586266 21105 leader_election.cc:288] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:06:02.586395 21109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:06:02.587520 21109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [term 2 LEADER]: Becoming Leader. State: Replica: 22898450805144a5a1673207360c9110, State: Running, Role: LEADER
I20260620 23:06:02.588124 21109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } }
I20260620 23:06:02.588406 21105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:06:02.590381 21111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22898450805144a5a1673207360c9110. Latest consensus state: current_term: 2 leader_uuid: "22898450805144a5a1673207360c9110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } } }
I20260620 23:06:02.590723 21111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:02.591176 21110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "22898450805144a5a1673207360c9110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22898450805144a5a1673207360c9110" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34119 } } }
I20260620 23:06:02.591464 21110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:02.596097 21116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:06:02.602234 21116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:06:02.603753 21116 catalog_manager.cc:1259] Loaded cluster ID: eceab26f75cd46de8525b81d43510161
I20260620 23:06:02.603941 21116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:06:02.607980 21116 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:06:02.609532 21116 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 22898450805144a5a1673207360c9110: Loaded TSK: 0
I20260620 23:06:02.611248 21116 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:06:02.891023 21107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:02.891304 21107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:02.891422 21107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:02.898778 21107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:02.898896 21107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:02.899067 21107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:06:02.908592 21107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.65.keytab
--principal=oryx/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:32881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=43437
--tserver_master_addrs=127.12.2.126:34119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:02.909884 21107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:02.911185 21107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:02.923776 21132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:02.924116 21134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:02.924217 21131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:02.924441 21107 server_base.cc:1034] running on GCE node
I20260620 23:06:02.925031 21107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:02.926011 21107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:02.927207 21107 hybrid_clock.cc:648] HybridClock initialized: now 1781996762927159 us; error 42 us; skew 500 ppm
I20260620 23:06:02.927503 21107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:06:02 dist-test-slave-7g8g krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996762, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:02.930516 21107 init.cc:377] Logged in from keytab as oryx/127.12.2.65@KRBTEST.COM (short username oryx)
I20260620 23:06:02.931927 21107 webserver.cc:460] Webserver started at http://127.12.2.65:43437/ using document root <none> and password file <none>
I20260620 23:06:02.932559 21107 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:02.932678 21107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:02.937800 21107 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:06:02.940554 21141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:02.941880 21107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:06:02.942014 21107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "1afcac0e51c34a31b308ca41528d6fc4"
format_stamp: "Formatted at 2026-06-20 23:06:00 on dist-test-slave-7g8g"
server_key: "82267c7cccd32637da02b48a0ace74d8"
server_key_iv: "29d77b2d488e85354b77fed1986f7828"
server_key_version: "encryptionkey@0"
I20260620 23:06:02.942889 21107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:02.956746 21107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:02.960762 21107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:02.960997 21107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:02.962008 21107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:02.963337 21107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:02.963419 21107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:02.963514 21107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:02.963581 21107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:02.991797 21107 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:32881
I20260620 23:06:02.991905 21253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:32881 every 8 connection(s)
I20260620 23:06:02.993198 21107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:06:03.000375 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 21107
I20260620 23:06:03.000932 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:41065
--local_ip_for_outbound_sockets=127.12.2.66
--tserver_master_addrs=127.12.2.126:34119
--webserver_port=36211
--webserver_interface=127.12.2.66
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.2.66
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:06:03.010826 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:02.995920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:34437 (local address 127.12.2.126:34119)
0620 23:06:02.996546 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:02.996558 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:02.996981 (+ 423us) server_negotiation.cc:407] Connection header received
0620 23:06:02.999063 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:02.999076 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:02.999407 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:02.999700 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.001189 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.002421 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.004009 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.004675 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.004845 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.009823 (+ 4978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:34437: BlockingRecv error: recv got EOF from 127.12.2.65:34437 (error 108)
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":220,"threads_started":1}
I20260620 23:06:03.010807 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:02.996041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:34437)
0620 23:06:02.996821 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:02.996865 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:02.998557 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:02.999838 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:02.999846 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.000355 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.000964 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.000983 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.002584 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.002589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.003867 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.003880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.004912 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.004919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.005171 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.005841 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.005863 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.009440 (+ 3577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":340,"threads_started":1}
W20260620 23:06:03.011646 21254 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:34119 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.2.126:34119: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:03.019572 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.012202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:33069)
0620 23:06:03.012361 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.012380 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.012469 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.012826 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.012831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.013148 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.013289 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.013297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.014271 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.014277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.015232 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.015245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.015872 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.015878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.016037 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.016590 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.016606 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.019296 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 23:06:03.019670 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.012425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:33069 (local address 127.12.2.126:34119)
0620 23:06:03.012589 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.012593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.012606 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:06:03.012647 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.012651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.012702 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.012787 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.013412 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.014137 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.015353 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.015749 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.015849 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.019483 (+ 3634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:33069: BlockingRecv error: recv got EOF from 127.12.2.65:33069 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:03.027570 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.020468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:47731)
0620 23:06:03.020616 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.020633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.020720 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.021000 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.021005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.021272 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.021405 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.021412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.022336 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.022340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.023247 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.023258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.023872 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.023878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.024041 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.024532 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.024551 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.027296 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 23:06:03.027742 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.020569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:47731 (local address 127.12.2.126:34119)
0620 23:06:03.020711 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.020716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.020783 (+ 67us) server_negotiation.cc:407] Connection header received
0620 23:06:03.020826 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.020829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.020892 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.020963 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.021524 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.022209 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.023363 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.023751 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.023850 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.027500 (+ 3650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:47731: BlockingRecv error: recv got EOF from 127.12.2.65:47731 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 23:06:03.027993 21254 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:06:03.316455 21258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:03.316725 21258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:03.316826 21258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:03.324095 21258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:03.324198 21258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:03.324316 21258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:06:03.333894 21258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.66.keytab
--principal=oryx/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:41065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=36211
--tserver_master_addrs=127.12.2.126:34119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:03.335183 21258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:03.336407 21258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:03.348915 21263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:03.348906 21266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:03.348946 21264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:03.350013 21258 server_base.cc:1034] running on GCE node
I20260620 23:06:03.350634 21258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:03.351665 21258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:03.352895 21258 hybrid_clock.cc:648] HybridClock initialized: now 1781996763352845 us; error 34 us; skew 500 ppm
I20260620 23:06:03.353222 21258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:03.356177 21258 init.cc:377] Logged in from keytab as oryx/127.12.2.66@KRBTEST.COM (short username oryx)
I20260620 23:06:03.357510 21258 webserver.cc:460] Webserver started at http://127.12.2.66:36211/ using document root <none> and password file <none>
I20260620 23:06:03.358194 21258 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:03.358320 21258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:03.363621 21258 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:06:03.366381 21273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:03.367797 21258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:06:03.367937 21258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "e7359bb69ef04494bc2527555b6eeadf"
format_stamp: "Formatted at 2026-06-20 23:06:00 on dist-test-slave-7g8g"
server_key: "9d13e7ea183e80851fb6a0400cf7b453"
server_key_iv: "f1ab08cf3cbb04e3e123e6f04134b746"
server_key_version: "encryptionkey@0"
I20260620 23:06:03.368870 21258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:03.388996 21258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:03.393090 21258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:03.393323 21258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:03.394228 21258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:03.395579 21258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:03.395678 21258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:03.395792 21258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:03.395859 21258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:03.424978 21258 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:41065
I20260620 23:06:03.425024 21385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:41065 every 8 connection(s)
I20260620 23:06:03.426352 21258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:06:03.431785 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 21258
I20260620 23:06:03.432467 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:39847
--local_ip_for_outbound_sockets=127.12.2.67
--tserver_master_addrs=127.12.2.126:34119
--webserver_port=41995
--webserver_interface=127.12.2.67
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.2.67
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:06:03.442629 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.428794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:49769 (local address 127.12.2.126:34119)
0620 23:06:03.428954 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.428959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.430045 (+ 1086us) server_negotiation.cc:407] Connection header received
0620 23:06:03.432116 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.432121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.432191 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.432281 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.434249 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.435037 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.437086 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.437515 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.437637 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.442404 (+ 4767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:49769: BlockingRecv error: recv got EOF from 127.12.2.66:49769 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:06:03.443235 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.429219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:49769)
0620 23:06:03.429887 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.429930 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.431769 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.432568 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.432580 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.433251 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.434042 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.434065 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.435195 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.435200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.436921 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.436935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.437759 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.437766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.438023 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.438705 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.438737 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.442170 (+ 3433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":259,"threads_started":1}
W20260620 23:06:03.443898 21386 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:34119 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.2.126:34119: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:03.451761 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.444430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:57463)
0620 23:06:03.444575 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.444593 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.444686 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.445006 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.445010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.445307 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.445442 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.445450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.446423 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.446428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.447370 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.447381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.448325 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.448330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.448450 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.448999 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.449016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.451563 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 23:06:03.452059 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.444592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:57463 (local address 127.12.2.126:34119)
0620 23:06:03.444759 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.444764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.444780 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:06:03.444836 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.444839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.444893 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.444969 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.445569 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.446295 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.447526 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.448163 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.448315 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.451822 (+ 3507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:57463: BlockingRecv error: recv got EOF from 127.12.2.66:57463 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:03.459941 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.452487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:43115)
0620 23:06:03.452656 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.452675 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.452857 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.453179 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.453183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.453453 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.453588 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.453596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.454518 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.454523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.455482 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.455494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.456131 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.456136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.456305 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.456874 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.456890 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.459599 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":298,"threads_started":1}
W20260620 23:06:03.460002 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.452486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:43115 (local address 127.12.2.126:34119)
0620 23:06:03.452655 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.452661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.452748 (+ 87us) server_negotiation.cc:407] Connection header received
0620 23:06:03.453004 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.453009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.453066 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.453144 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.453714 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.454396 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.455606 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.456004 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.456106 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.459809 (+ 3703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:43115: BlockingRecv error: recv got EOF from 127.12.2.66:43115 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 23:06:03.460306 21386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:06:03.747699 21389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:03.748021 21389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:03.748157 21389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:03.755203 21389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:03.755329 21389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:03.755470 21389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:06:03.765125 21389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/krb5kdc/oryx_127.12.2.67.keytab
--principal=oryx/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:39847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=41995
--tserver_master_addrs=127.12.2.126:34119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:03.766405 21389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:03.767683 21389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:03.779728 21396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:03.779862 21395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:03.779800 21398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:03.780856 21389 server_base.cc:1034] running on GCE node
I20260620 23:06:03.781459 21389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:03.782366 21389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:03.783577 21389 hybrid_clock.cc:648] HybridClock initialized: now 1781996763783532 us; error 37 us; skew 500 ppm
I20260620 23:06:03.783898 21389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:03.786974 21389 init.cc:377] Logged in from keytab as oryx/127.12.2.67@KRBTEST.COM (short username oryx)
I20260620 23:06:03.788311 21389 webserver.cc:460] Webserver started at http://127.12.2.67:41995/ using document root <none> and password file <none>
I20260620 23:06:03.789022 21389 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:03.789149 21389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:03.794343 21389 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:06:03.797011 21405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:03.798312 21389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:06:03.798441 21389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "3004aabf838342df95c604b07d4627d8"
format_stamp: "Formatted at 2026-06-20 23:06:01 on dist-test-slave-7g8g"
server_key: "a1c22baadbae438bfc8426d53e1edaee"
server_key_iv: "3c55c933461cb4b9e4e131affcc0241c"
server_key_version: "encryptionkey@0"
I20260620 23:06:03.799367 21389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:03.815375 21389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:03.819190 21389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:03.819427 21389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:03.820336 21389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:03.821542 21389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:03.821635 21389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:03.821744 21389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:03.821815 21389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:03.850458 21389 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:39847
I20260620 23:06:03.850510 21517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:39847 every 8 connection(s)
I20260620 23:06:03.851874 21389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:06:03.854364 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 21389
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:06:03.867591 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.854408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:36465 (local address 127.12.2.126:34119)
0620 23:06:03.854604 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.854610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.856008 (+ 1398us) server_negotiation.cc:407] Connection header received
0620 23:06:03.858065 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.858071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.858152 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.858261 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.859651 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.860447 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.861890 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.862285 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.862387 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.867391 (+ 5004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:36465: BlockingRecv error: recv got EOF from 127.12.2.67:36465 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:06:03.868337 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.854925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:36465)
0620 23:06:03.855836 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.855889 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.857820 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.858451 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.858458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.858913 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.859498 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.859515 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.860585 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.860589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.861765 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.861776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.862483 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.862490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.862697 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.863324 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.863345 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.867221 (+ 3876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":340,"threads_started":1}
W20260620 23:06:03.868989 21518 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:34119 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.2.126:34119: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
I20260620 23:06:03.871012 21522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.855304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.12.2.126:34119)
0620 23:06:03.856571 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.856574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.857010 (+ 436us) server_negotiation.cc:407] Connection header received
0620 23:06:03.857170 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.857175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.857244 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.857330 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.858876 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.859636 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.860892 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.861288 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.861387 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.863855 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.863863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.863878 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.863910 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.867612 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.868382 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.868390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.868399 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.868582 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.869036 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.869040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.869043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.869561 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.869856 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.870380 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.870622 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1179,"thread_start_us":1082,"threads_started":1}
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:03.877122 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.869530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:47265)
0620 23:06:03.869685 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.869702 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.869776 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.870082 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.870087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.870471 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.870640 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.870651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.871954 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.871958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.872885 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.872897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.873687 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.873692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.873906 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.874404 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.874426 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.876932 (+ 2506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 23:06:03.877291 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.869622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:47265 (local address 127.12.2.126:34119)
0620 23:06:03.869845 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.869850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.869866 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:06:03.869906 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.869909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.869956 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.870024 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.870813 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.871808 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.873009 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.873551 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.873672 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.877107 (+ 3435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:47265: BlockingRecv error: recv got EOF from 127.12.2.67:47265 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:03 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:03.884881 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.877890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:44119)
0620 23:06:03.878061 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.878079 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.878165 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.878434 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.878438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.878734 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.878874 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.878882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.879980 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.879984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.880880 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.880890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.881503 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.881507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.881647 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.882172 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.882188 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.884617 (+ 2429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:06:03.884967 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.877978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:44119 (local address 127.12.2.126:34119)
0620 23:06:03.878176 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.878180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.878195 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:06:03.878251 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.878255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.878301 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.878369 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.879063 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.879853 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.881019 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.881381 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.881479 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.884778 (+ 3299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:44119: BlockingRecv error: recv got EOF from 127.12.2.67:44119 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 23:06:03.885443 21518 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 23:06:04 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:04 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:04.036612 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:04.028638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:36795)
0620 23:06:04.029161 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:04.029180 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:04.029331 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:04.029696 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:04.029699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:04.030196 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:04.030433 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:04.030445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.031438 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.031442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:04.032400 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:04.032413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.033072 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.033076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:04.033198 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:04.033863 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:04.033881 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:04.036392 (+ 2511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":311,"threads_started":1}
W20260620 23:06:04.036863 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:04.028755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:36795 (local address 127.12.2.126:34119)
0620 23:06:04.028965 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:04.028971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:04.029296 (+ 325us) server_negotiation.cc:407] Connection header received
0620 23:06:04.029458 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:04.029462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:04.029531 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:04.029634 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:04.030557 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.031290 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.032512 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.032924 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.033008 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:04.036613 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:36795: BlockingRecv error: recv got EOF from 127.12.2.65:36795 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 23:06:04 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:04 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:04.469547 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:04.460887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:43069)
0620 23:06:04.461383 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:04.461401 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:04.461521 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:04.461901 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:04.461905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:04.462368 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:04.462568 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:04.462578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.463605 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.463609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:04.464608 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:04.464620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.465339 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.465343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:04.465461 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:04.466027 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:04.466046 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:04.469240 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":252,"threads_started":1}
W20260620 23:06:04.469667 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:04.460939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:43069 (local address 127.12.2.126:34119)
0620 23:06:04.461124 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:04.461130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:04.461470 (+ 340us) server_negotiation.cc:407] Connection header received
0620 23:06:04.461665 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:04.461669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:04.461732 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:04.461818 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:04.462720 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.463458 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.464768 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.465172 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.465290 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:04.469441 (+ 4151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:43069: BlockingRecv error: recv got EOF from 127.12.2.66:43069 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 23:06:04 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:04 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:04.894424 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:04.885994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:33273)
0620 23:06:04.886490 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:04.886509 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:04.886633 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:04.886961 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:04.886967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:04.887449 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:04.887656 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:04.887666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.888773 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.888777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:04.889859 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:04.889876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.890606 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.890612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:04.890737 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:04.891433 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:04.891451 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:04.894187 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":262,"threads_started":1}
W20260620 23:06:04.894565 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:04.886076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:33273 (local address 127.12.2.126:34119)
0620 23:06:04.886315 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:04.886322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:04.886628 (+ 306us) server_negotiation.cc:407] Connection header received
0620 23:06:04.886730 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:04.886734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:04.886802 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:04.886882 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:04.887779 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.888621 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.890014 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.890422 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.890517 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:04.894373 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:33273: BlockingRecv error: recv got EOF from 127.12.2.67:33273 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 23:06:05 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:05 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:05.045539 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:05.037478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:39743)
0620 23:06:05.037997 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:05.038013 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:05.038133 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:05.038419 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:05.038422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:05.038784 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:05.038974 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:05.038984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.040327 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.040331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:05.041332 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:05.041345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.041977 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.041983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:05.042121 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:05.042644 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:05.042661 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:05.045221 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":284,"threads_started":1}
W20260620 23:06:05.045658 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:05.037549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:39743 (local address 127.12.2.126:34119)
0620 23:06:05.037700 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:05.037705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:05.038076 (+ 371us) server_negotiation.cc:407] Connection header received
0620 23:06:05.038241 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:05.038246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:05.038304 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:05.038387 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:05.039109 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.040179 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.041459 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.041837 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.041944 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:05.045432 (+ 3488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:39743: BlockingRecv error: recv got EOF from 127.12.2.65:39743 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 23:06:05 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:05 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:05.478785 21527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:05.470921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:60569)
0620 23:06:05.471426 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:05.471442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:05.471563 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:05.471946 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:05.471951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:05.472350 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:05.472536 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:05.472546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.473515 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.473519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:05.474399 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:05.474410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.475101 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.475107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:05.475259 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:05.475806 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:05.475823 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:05.478419 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":247,"threads_started":1}
W20260620 23:06:05.478814 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:05.471023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:60569 (local address 127.12.2.126:34119)
0620 23:06:05.471167 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:05.471171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:05.471504 (+ 333us) server_negotiation.cc:407] Connection header received
0620 23:06:05.471716 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:05.471721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:05.471789 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:05.471872 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:05.472665 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.473382 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.474525 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.474967 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.475078 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:05.478610 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:60569: BlockingRecv error: recv got EOF from 127.12.2.66:60569 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 23:06:05 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:05 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:05.903506 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:05.895342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:60833)
0620 23:06:05.895811 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:05.895828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:05.895965 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:05.896336 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:05.896340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:05.896715 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:05.896872 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:05.896881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.897974 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.897979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:05.898901 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:05.898913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.899621 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.899626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:05.899741 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:05.900276 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:05.900292 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:05.903145 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":256,"threads_started":1}
W20260620 23:06:05.903720 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:05.895507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:60833 (local address 127.12.2.126:34119)
0620 23:06:05.895654 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:05.895658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:05.895899 (+ 241us) server_negotiation.cc:407] Connection header received
0620 23:06:05.896100 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:05.896104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:05.896170 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:05.896258 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:05.897023 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.897837 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.899056 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:05.899475 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:05.899568 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:05.903356 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:60833: BlockingRecv error: recv got EOF from 127.12.2.67:60833 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 23:06:06 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:06 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:06.055433 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.046605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:51595)
0620 23:06:06.047195 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.047215 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.047337 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.047643 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.047646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.048021 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.048186 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.048194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.049400 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.049404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.050521 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.050545 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.051352 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.051357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.051493 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.052163 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.052181 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.055073 (+ 2892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":364,"threads_started":1}
W20260620 23:06:06.055526 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.046659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:51595 (local address 127.12.2.126:34119)
0620 23:06:06.046850 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.046854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.047285 (+ 431us) server_negotiation.cc:407] Connection header received
0620 23:06:06.047450 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.047454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.047518 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.047593 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.048348 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.049221 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.050731 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.051201 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.051303 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.055286 (+ 3983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:51595: BlockingRecv error: recv got EOF from 127.12.2.65:51595 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 23:06:06 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:06 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:06.490741 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.479849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:49705)
0620 23:06:06.480330 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.480348 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.480471 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.480792 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.480796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.481151 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.481322 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.481332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.482302 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.482306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.483416 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.483431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.484185 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.484192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.484360 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.487719 (+ 3359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.487737 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.490418 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":253,"threads_started":1}
W20260620 23:06:06.490867 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.479978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:49705 (local address 127.12.2.126:34119)
0620 23:06:06.480150 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.480155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.480417 (+ 262us) server_negotiation.cc:407] Connection header received
0620 23:06:06.480568 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.480572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.480630 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.480714 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.481456 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.482204 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.483546 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.484028 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.484145 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.490647 (+ 6502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:49705: BlockingRecv error: recv got EOF from 127.12.2.66:49705 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 23:06:06 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:06 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:06.912182 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.904313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:35597)
0620 23:06:06.904834 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.904850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.904988 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.905312 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.905317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.905653 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.905812 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.905821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.906793 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.906798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.907744 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.907758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.908413 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.908419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.908580 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.909152 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.909169 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.911941 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":260,"threads_started":1}
W20260620 23:06:06.912281 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.904420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:35597 (local address 127.12.2.126:34119)
0620 23:06:06.904601 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.904607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.904931 (+ 324us) server_negotiation.cc:407] Connection header received
0620 23:06:06.905101 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.905107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.905174 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.905252 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.905951 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.906658 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.907879 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.908281 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.908389 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.912113 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:35597: BlockingRecv error: recv got EOF from 127.12.2.67:35597 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 23:06:07 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:07 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:07.063593 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:07.056281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:39435)
0620 23:06:07.056715 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:07.056732 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:07.056843 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:07.057090 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:07.057092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:07.057426 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:07.057585 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:07.057594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.058522 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.058525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:07.059506 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:07.059518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.060153 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.060157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:07.060276 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:07.060782 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:07.060799 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:07.063368 (+ 2569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":225,"threads_started":1}
W20260620 23:06:07.063776 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:07.056353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:39435 (local address 127.12.2.126:34119)
0620 23:06:07.056568 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:07.056573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:07.056851 (+ 278us) server_negotiation.cc:407] Connection header received
0620 23:06:07.056924 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:07.056928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:07.056988 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:07.057064 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:07.057696 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.058422 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.059623 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.060047 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.060139 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:07.063566 (+ 3427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:39435: BlockingRecv error: recv got EOF from 127.12.2.65:39435 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 23:06:07 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:07 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:07.499990 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:07.491751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:44701)
0620 23:06:07.492232 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:07.492248 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:07.492361 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:07.492683 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:07.492686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:07.493050 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:07.493192 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:07.493201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.494106 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.494110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:07.495101 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:07.495113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.495848 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.495853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:07.496037 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:07.496686 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:07.496708 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:07.499770 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":242,"threads_started":1}
W20260620 23:06:07.500228 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:07.491786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:44701 (local address 127.12.2.126:34119)
0620 23:06:07.491934 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:07.491940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:07.492319 (+ 379us) server_negotiation.cc:407] Connection header received
0620 23:06:07.492488 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:07.492492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:07.492569 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:07.492656 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:07.493319 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.494010 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.495238 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.495684 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.495789 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:07.499962 (+ 4173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:44701: BlockingRecv error: recv got EOF from 127.12.2.66:44701 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 23:06:07 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:07 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:07.920676 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:07.913060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:33553)
0620 23:06:07.913480 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:07.913497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:07.913615 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:07.913870 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:07.913873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:07.914206 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:07.914350 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:07.914358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.915347 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.915352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:07.916333 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:07.916344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.916981 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.916985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:07.917112 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:07.917705 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:07.917723 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:07.920369 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":225,"threads_started":1}
W20260620 23:06:07.920804 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:07.913174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:33553 (local address 127.12.2.126:34119)
0620 23:06:07.913411 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:07.913418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:07.913565 (+ 147us) server_negotiation.cc:407] Connection header received
0620 23:06:07.913701 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:07.913705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:07.913765 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:07.913847 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:07.914457 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.915230 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.916443 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.916866 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.916954 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:07.920586 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:33553: BlockingRecv error: recv got EOF from 127.12.2.67:33553 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 23:06:08 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:08 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:08.072744 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:08.064511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:34019)
0620 23:06:08.065151 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:08.065167 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:08.065294 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:08.065609 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:08.065614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:08.066049 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:08.066205 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:08.066214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.067257 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.067261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:08.068159 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:08.068172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.068880 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.068886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:08.069030 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:08.069601 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:08.069619 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:08.072372 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":379,"threads_started":1}
W20260620 23:06:08.072820 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:08.064639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:34019 (local address 127.12.2.126:34119)
0620 23:06:08.064884 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:08.064891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:08.065253 (+ 362us) server_negotiation.cc:407] Connection header received
0620 23:06:08.065410 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:08.065416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:08.065479 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:08.065560 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:08.066366 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.067123 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.068296 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.068714 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.068820 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:08.072595 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:34019: BlockingRecv error: recv got EOF from 127.12.2.65:34019 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 20 23:06:08 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:08 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:08.508637 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:08.500805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:59651)
0620 23:06:08.501292 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:08.501308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:08.501425 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:08.501746 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:08.501749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:08.502136 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:08.502297 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:08.502306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.503461 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.503466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:08.504408 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:08.504420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.505091 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.505095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:08.505207 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:08.505732 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:08.505748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:08.508383 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":237,"threads_started":1}
W20260620 23:06:08.508838 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:08.501002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:59651 (local address 127.12.2.126:34119)
0620 23:06:08.501172 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:08.501176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:08.501376 (+ 200us) server_negotiation.cc:407] Connection header received
0620 23:06:08.501556 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:08.501559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:08.501626 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:08.501727 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:08.502455 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.503289 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.504542 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.504982 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.505071 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:08.508612 (+ 3541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:59651: BlockingRecv error: recv got EOF from 127.12.2.66:59651 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 23:06:08 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:08 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:08.929296 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:08.921537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:50837)
0620 23:06:08.922030 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:08.922047 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:08.922166 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:08.922482 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:08.922486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:08.922903 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:08.923112 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:08.923122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.924059 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.924064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:08.924959 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:08.924970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.925640 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.925646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:08.925805 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:08.926346 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:08.926361 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:08.928977 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":219,"threads_started":1}
W20260620 23:06:08.929389 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:08.921627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:50837 (local address 127.12.2.126:34119)
0620 23:06:08.921799 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:08.921809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:08.922129 (+ 320us) server_negotiation.cc:407] Connection header received
0620 23:06:08.922268 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:08.922273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:08.922352 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:08.922427 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:08.923246 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.923931 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.925093 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:08.925512 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:08.925615 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:08.929173 (+ 3558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:50837: BlockingRecv error: recv got EOF from 127.12.2.67:50837 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 23:06:09 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:09 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:09.081259 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:09.073710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:38575)
0620 23:06:09.074204 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:09.074219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:09.074361 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:09.074670 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:09.074675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:09.075096 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:09.075255 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:09.075264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.076246 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.076251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:09.077123 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:09.077135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.077798 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.077804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:09.077961 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:09.078484 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:09.078501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:09.081048 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":242,"threads_started":1}
W20260620 23:06:09.081485 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:09.073789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:38575 (local address 127.12.2.126:34119)
0620 23:06:09.073985 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:09.073990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:09.074298 (+ 308us) server_negotiation.cc:407] Connection header received
0620 23:06:09.074473 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:09.074477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:09.074542 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:09.074637 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:09.075391 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.076117 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.077259 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.077658 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.077777 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:09.081227 (+ 3450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:38575: BlockingRecv error: recv got EOF from 127.12.2.65:38575 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 23:06:09 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:09 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:09.517158 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:09.509516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:39767)
0620 23:06:09.509973 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:09.509989 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:09.510114 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:09.510429 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:09.510434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:09.510795 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:09.510981 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:09.510990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.511950 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.511956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:09.512827 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:09.512840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.513487 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.513493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:09.513642 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:09.514144 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:09.514161 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:09.516938 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":273,"threads_started":1}
W20260620 23:06:09.517360 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:09.509597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:39767 (local address 127.12.2.126:34119)
0620 23:06:09.509783 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:09.509789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:09.510073 (+ 284us) server_negotiation.cc:407] Connection header received
0620 23:06:09.510226 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:09.510231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:09.510293 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:09.510367 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:09.511120 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.511816 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.512961 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.513350 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.513452 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:09.517132 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:39767: BlockingRecv error: recv got EOF from 127.12.2.66:39767 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 23:06:09 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:09 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:09.938043 21540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:09.930219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:38791)
0620 23:06:09.930704 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:09.930720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:09.930848 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:09.931238 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:09.931243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:09.931702 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:09.931905 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:09.931914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.932897 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.932901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:09.933766 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:09.933787 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.934508 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.934513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:09.934664 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:09.935227 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:09.935244 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:09.937829 (+ 2585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":269,"threads_started":1}
W20260620 23:06:09.938202 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:09.930346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:38791 (local address 127.12.2.126:34119)
0620 23:06:09.930526 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:09.930531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:09.930796 (+ 265us) server_negotiation.cc:407] Connection header received
0620 23:06:09.930998 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:09.931003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:09.931078 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:09.931191 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:09.932046 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.932763 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.933911 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:09.934365 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:09.934469 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:09.937991 (+ 3522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:38791: BlockingRecv error: recv got EOF from 127.12.2.67:38791 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 23:06:10 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:10 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:10.090132 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:10.082087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:54037)
0620 23:06:10.082582 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:10.082598 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:10.082708 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:10.083021 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:10.083024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:10.083405 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:10.083566 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:10.083575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.084538 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.084542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:10.085714 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:10.085729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.086546 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.086551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:10.086664 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:10.087259 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:10.087276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:10.089887 (+ 2611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":222,"threads_started":1}
W20260620 23:06:10.090246 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:10.082194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:54037 (local address 127.12.2.126:34119)
0620 23:06:10.082411 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:10.082417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:10.082670 (+ 253us) server_negotiation.cc:407] Connection header received
0620 23:06:10.082826 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:10.082829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:10.082890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:10.082998 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:10.083680 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.084398 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.085840 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.086417 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.086508 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:10.090062 (+ 3554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:54037: BlockingRecv error: recv got EOF from 127.12.2.65:54037 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 23:06:10 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:10 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:10.526072 21542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:10.518086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:38731)
0620 23:06:10.518595 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:10.518611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:10.518723 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:10.519021 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:10.519023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:10.519391 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:10.519559 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:10.519568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.520560 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.520563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:10.521492 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:10.521503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.522082 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.522086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:10.522194 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:10.522748 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:10.522765 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:10.525827 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":225,"threads_started":1}
W20260620 23:06:10.526209 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:10.518135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:38731 (local address 127.12.2.126:34119)
0620 23:06:10.518321 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:10.518328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:10.518685 (+ 357us) server_negotiation.cc:407] Connection header received
0620 23:06:10.518819 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:10.518822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:10.518909 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:10.519005 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:10.519697 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.520440 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.521595 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.521962 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.522047 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:10.526011 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:38731: BlockingRecv error: recv got EOF from 127.12.2.66:38731 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 23:06:10 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:10 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:10.946580 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:10.938906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:49377)
0620 23:06:10.939427 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:10.939443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:10.939558 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:10.939838 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:10.939841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:10.940263 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:10.940458 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:10.940468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.941581 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.941587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:10.942513 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:10.942525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.943215 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.943219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:10.943325 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:10.943815 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:10.943830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:10.946338 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":200,"threads_started":1}
W20260620 23:06:10.946672 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:10.938954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:49377 (local address 127.12.2.126:34119)
0620 23:06:10.939107 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:10.939111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:10.939526 (+ 415us) server_negotiation.cc:407] Connection header received
0620 23:06:10.939647 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:10.939651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:10.939713 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:10.939790 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:10.940581 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.941447 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.942640 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:10.943106 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:10.943197 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:10.946508 (+ 3311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:49377: BlockingRecv error: recv got EOF from 127.12.2.67:49377 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 23:06:11 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:11 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:11.099223 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:11.091027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:51155)
0620 23:06:11.091513 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:11.091528 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:11.091645 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:11.091933 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:11.091937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:11.092418 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:11.092565 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:11.092574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.093525 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.093531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:11.094480 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:11.094492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.095255 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.095262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:11.095435 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:11.096065 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:11.096082 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:11.098981 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":227,"threads_started":1}
W20260620 23:06:11.099330 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:11.091199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:51155 (local address 127.12.2.126:34119)
0620 23:06:11.091381 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:11.091388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:11.091594 (+ 206us) server_negotiation.cc:407] Connection header received
0620 23:06:11.091760 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:11.091763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:11.091820 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:11.091924 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:11.092692 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.093392 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.094607 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.095098 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.095204 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:11.099136 (+ 3932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:51155: BlockingRecv error: recv got EOF from 127.12.2.65:51155 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 23:06:11 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:11 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:11.539500 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:11.527014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:56391)
0620 23:06:11.527517 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:11.527533 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:11.527672 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:11.528016 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:11.528021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:11.528413 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:11.528563 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:11.528572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.529802 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.529812 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:11.531006 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:11.531021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.531988 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.531993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:11.532110 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:11.536543 (+ 4433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:11.536563 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:11.539281 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":250,"threads_started":1}
W20260620 23:06:11.539726 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:11.527100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:56391 (local address 127.12.2.126:34119)
0620 23:06:11.527277 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:11.527284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:11.527627 (+ 343us) server_negotiation.cc:407] Connection header received
0620 23:06:11.527797 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:11.527801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:11.527872 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:11.527971 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:11.528711 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.529665 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.531127 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.531819 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.531938 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:11.539507 (+ 7569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:56391: BlockingRecv error: recv got EOF from 127.12.2.66:56391 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 23:06:11 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:11 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:11.955649 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:11.947571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:47203)
0620 23:06:11.948097 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:11.948115 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:11.948237 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:11.948554 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:11.948557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:11.948999 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:11.949155 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:11.949165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.950218 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.950223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:11.951223 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:11.951236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.951950 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.951955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:11.952082 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:11.952655 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:11.952672 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:11.955340 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":259,"threads_started":1}
W20260620 23:06:11.955761 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:11.947606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:47203 (local address 127.12.2.126:34119)
0620 23:06:11.947776 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:11.947781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:11.948222 (+ 441us) server_negotiation.cc:407] Connection header received
0620 23:06:11.948341 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:11.948345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:11.948408 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:11.948489 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:11.949305 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.950066 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.951360 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:11.951781 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:11.951883 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:11.955551 (+ 3668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:47203: BlockingRecv error: recv got EOF from 127.12.2.67:47203 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 23:06:12 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:12 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:12.108189 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:12.100143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:59135)
0620 23:06:12.100653 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:12.100669 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:12.100797 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:12.101124 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:12.101128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:12.101526 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:12.101682 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:12.101691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.102696 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.102701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:12.103657 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:12.103669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.104329 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.104335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:12.104494 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:12.105024 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:12.105040 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:12.107861 (+ 2821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":276,"threads_started":1}
W20260620 23:06:12.108297 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:12.100208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:59135 (local address 127.12.2.126:34119)
0620 23:06:12.100387 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:12.100393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:12.100750 (+ 357us) server_negotiation.cc:407] Connection header received
0620 23:06:12.100911 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:12.100916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:12.100995 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:12.101072 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:12.101813 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.102575 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.103790 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.104193 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.104295 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:12.108081 (+ 3786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:59135: BlockingRecv error: recv got EOF from 127.12.2.65:59135 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 23:06:12 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:12 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:12.548470 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:12.540346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:37329)
0620 23:06:12.541011 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:12.541031 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:12.541156 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:12.541480 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:12.541484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:12.541936 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:12.542094 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:12.542103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.543100 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.543105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:12.544069 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:12.544080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.544744 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.544750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:12.544917 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:12.545420 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:12.545437 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:12.548141 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":395,"threads_started":1}
W20260620 23:06:12.548560 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:12.540427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:37329 (local address 127.12.2.126:34119)
0620 23:06:12.540604 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:12.540611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:12.541107 (+ 496us) server_negotiation.cc:407] Connection header received
0620 23:06:12.541270 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:12.541275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:12.541354 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:12.541435 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:12.542231 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.542977 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.544194 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.544604 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.544704 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:12.548349 (+ 3645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:37329: BlockingRecv error: recv got EOF from 127.12.2.66:37329 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 23:06:12 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:12 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:12.964515 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:12.956625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:44059)
0620 23:06:12.957112 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:12.957128 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:12.957243 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:12.957572 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:12.957576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:12.957956 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:12.958122 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:12.958134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.959163 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.959167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:12.960206 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:12.960220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.960947 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.960952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:12.961122 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:12.961639 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:12.961656 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:12.964209 (+ 2553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":267,"threads_started":1}
W20260620 23:06:12.964558 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:12.956734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:44059 (local address 127.12.2.126:34119)
0620 23:06:12.956963 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:12.956969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:12.957210 (+ 241us) server_negotiation.cc:407] Connection header received
0620 23:06:12.957336 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:12.957340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:12.957405 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:12.957491 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:12.958252 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.959035 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.960329 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:12.960798 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:12.960913 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:12.964387 (+ 3474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:44059: BlockingRecv error: recv got EOF from 127.12.2.67:44059 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 23:06:13 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:13 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:13.117532 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:13.109337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:53817)
0620 23:06:13.109784 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:13.109837 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:13.109947 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:13.110201 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:13.110204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:13.110551 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:13.110700 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:13.110708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.111906 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.111911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:13.112850 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:13.112862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.113559 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.113563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:13.113689 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:13.114288 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:13.114309 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:13.117217 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":247,"threads_started":1}
W20260620 23:06:13.117676 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:13.109368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:53817 (local address 127.12.2.126:34119)
0620 23:06:13.109517 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:13.109522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:13.109897 (+ 375us) server_negotiation.cc:407] Connection header received
0620 23:06:13.110027 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:13.110030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:13.110106 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:13.110181 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:13.110881 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.111711 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.112979 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.113426 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.113517 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:13.117440 (+ 3923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:53817: BlockingRecv error: recv got EOF from 127.12.2.65:53817 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 23:06:13 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:13 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:13.557343 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:13.549477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:35147)
0620 23:06:13.549922 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:13.549939 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:13.550046 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:13.550300 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:13.550303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:13.550650 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:13.550831 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:13.550841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.551761 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.551765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:13.552718 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:13.552730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.553380 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.553384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:13.553504 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:13.554171 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:13.554196 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:13.557135 (+ 2939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":240,"threads_started":1}
W20260620 23:06:13.557590 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:13.549542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:35147 (local address 127.12.2.126:34119)
0620 23:06:13.549697 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:13.549702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:13.550000 (+ 298us) server_negotiation.cc:407] Connection header received
0620 23:06:13.550135 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:13.550138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:13.550193 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:13.550268 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:13.550966 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.551637 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.552825 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.553256 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.553339 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:13.557344 (+ 4005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:35147: BlockingRecv error: recv got EOF from 127.12.2.66:35147 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 23:06:13 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:13 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:13.973701 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:13.965489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:45333)
0620 23:06:13.965988 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:13.966004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:13.966111 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:13.966385 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:13.966388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:13.966714 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:13.966879 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:13.966888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.967884 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.967888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:13.968797 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:13.968808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.969454 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.969459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:13.969606 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:13.970242 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:13.970262 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:13.973379 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":260,"threads_started":1}
W20260620 23:06:13.973845 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:13.965656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:45333 (local address 127.12.2.126:34119)
0620 23:06:13.965858 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:13.965863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:13.966074 (+ 211us) server_negotiation.cc:407] Connection header received
0620 23:06:13.966195 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:13.966198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:13.966267 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:13.966352 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:13.967009 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.967726 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.968914 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:13.969291 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:13.969389 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:13.973622 (+ 4233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:45333: BlockingRecv error: recv got EOF from 127.12.2.67:45333 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 20 23:06:14 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:14 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:14.126681 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:14.118500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:39763)
0620 23:06:14.119024 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:14.119041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:14.119173 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:14.119507 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:14.119510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:14.119879 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:14.120084 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:14.120093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.121187 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.121191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:14.122070 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:14.122082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.122830 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.122834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:14.123024 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:14.123610 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:14.123628 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:14.126347 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":239,"threads_started":1}
W20260620 23:06:14.126777 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:14.118592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:39763 (local address 127.12.2.126:34119)
0620 23:06:14.118774 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:14.118780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:14.119132 (+ 352us) server_negotiation.cc:407] Connection header received
0620 23:06:14.119292 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:14.119298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:14.119387 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:14.119478 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:14.120236 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.121045 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.122200 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.122692 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.122804 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:14.126558 (+ 3754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:39763: BlockingRecv error: recv got EOF from 127.12.2.65:39763 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:06:14 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:14 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:14.566177 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:14.558273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:57909)
0620 23:06:14.558795 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:14.558814 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:14.558969 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:14.559281 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:14.559285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:14.559680 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:14.559850 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:14.559860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.560833 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.560839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:14.561710 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:14.561720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.562411 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.562417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:14.562572 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:14.563172 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:14.563190 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:14.565860 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":301,"threads_started":1}
W20260620 23:06:14.566242 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:14.558377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:57909 (local address 127.12.2.126:34119)
0620 23:06:14.558561 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:14.558567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:14.558897 (+ 330us) server_negotiation.cc:407] Connection header received
0620 23:06:14.559088 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:14.559093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:14.559158 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:14.559234 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:14.559996 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.560705 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.561860 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.562274 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.562380 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:14.566031 (+ 3651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:57909: BlockingRecv error: recv got EOF from 127.12.2.66:57909 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 23:06:14 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:14 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:14.982857 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:14.974685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:45235)
0620 23:06:14.975190 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:14.975208 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:14.975323 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:14.975662 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:14.975666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:14.976134 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:14.976327 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:14.976337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.977390 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.977394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:14.978344 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:14.978356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.979098 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.979104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:14.979252 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:14.979865 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:14.979887 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:14.982541 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":289,"threads_started":1}
W20260620 23:06:14.982992 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:14.974792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:45235 (local address 127.12.2.126:34119)
0620 23:06:14.974997 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:14.975004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:14.975301 (+ 297us) server_negotiation.cc:407] Connection header received
0620 23:06:14.975416 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:14.975419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:14.975487 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:14.975574 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:14.976481 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.977254 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.978477 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:14.978898 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:14.979055 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:14.982732 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:45235: BlockingRecv error: recv got EOF from 127.12.2.67:45235 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 23:06:15 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:15 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:15.135634 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:15.127722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:56431)
0620 23:06:15.128193 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:15.128210 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:15.128334 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:15.128639 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:15.128643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:15.129067 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:15.129220 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:15.129229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.130202 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.130207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:15.131178 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:15.131191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.131877 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.131881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:15.132019 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:15.132572 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:15.132587 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:15.135359 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
W20260620 23:06:15.136044 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:15.127781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:56431 (local address 127.12.2.126:34119)
0620 23:06:15.128034 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:15.128041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:15.128287 (+ 246us) server_negotiation.cc:407] Connection header received
0620 23:06:15.128419 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:15.128422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:15.128502 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:15.128584 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:15.129348 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.130062 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.131281 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.131729 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.131816 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:15.135593 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:56431: BlockingRecv error: recv got EOF from 127.12.2.65:56431 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 23:06:15 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:15 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:15.575016 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:15.567100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:56387)
0620 23:06:15.567536 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:15.567553 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:15.567662 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:15.567921 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:15.567924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:15.568289 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:15.568436 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:15.568444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.569391 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.569394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:15.570294 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:15.570306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.571040 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.571045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:15.571172 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:15.571732 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:15.571749 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:15.574669 (+ 2920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":226,"threads_started":1}
W20260620 23:06:15.575124 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:15.567176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:56387 (local address 127.12.2.126:34119)
0620 23:06:15.567377 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:15.567383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:15.567623 (+ 240us) server_negotiation.cc:407] Connection header received
0620 23:06:15.567753 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:15.567756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:15.567814 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:15.567912 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:15.568559 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.569275 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.570418 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.570866 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.570993 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:15.574868 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:56387: BlockingRecv error: recv got EOF from 127.12.2.66:56387 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 23:06:15 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:15 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:15.992434 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:15.983884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:58821)
0620 23:06:15.984408 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:15.984424 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:15.984549 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:15.984840 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:15.984845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:15.985283 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:15.985438 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:15.985449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.986519 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.986523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:15.987472 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:15.987486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.988246 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.988250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:15.988454 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:15.989082 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:15.989099 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:15.992021 (+ 2922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":266,"threads_started":1}
W20260620 23:06:15.992594 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:15.984036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:58821 (local address 127.12.2.126:34119)
0620 23:06:15.984242 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:15.984248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:15.984513 (+ 265us) server_negotiation.cc:407] Connection header received
0620 23:06:15.984638 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:15.984641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:15.984704 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:15.984784 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:15.985629 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.986359 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.987622 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:15.988090 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:15.988198 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:15.992275 (+ 4077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:58821: BlockingRecv error: recv got EOF from 127.12.2.67:58821 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 23:06:16 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:16 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:16.144759 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:16.136525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:45039)
0620 23:06:16.137008 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:16.137024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:16.137139 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:16.137470 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:16.137474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:16.137872 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:16.138034 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:16.138044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.139159 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.139164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:16.140115 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:16.140129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.140931 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.140938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:16.141097 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:16.141712 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:16.141737 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:16.144533 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":245,"threads_started":1}
W20260620 23:06:16.144914 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:16.136618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:45039 (local address 127.12.2.126:34119)
0620 23:06:16.136781 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:16.136786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:16.137096 (+ 310us) server_negotiation.cc:407] Connection header received
0620 23:06:16.137243 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:16.137246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:16.137321 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:16.137396 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:16.138166 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.138879 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.140263 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.140727 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.140864 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:16.144707 (+ 3843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:45039: BlockingRecv error: recv got EOF from 127.12.2.65:45039 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 23:06:16 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:16 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:16.588106 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:16.575945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:40353)
0620 23:06:16.576402 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:16.576419 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:16.576531 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:16.576792 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:16.576795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:16.577188 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:16.577339 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:16.577348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.578345 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.578348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:16.579314 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:16.579326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.580056 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.580062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:16.580215 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:16.584697 (+ 4482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:16.584720 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:16.587748 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":222,"threads_started":1}
W20260620 23:06:16.588289 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:16.576143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:40353 (local address 127.12.2.126:34119)
0620 23:06:16.576342 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:16.576349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:16.576482 (+ 133us) server_negotiation.cc:407] Connection header received
0620 23:06:16.576623 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:16.576627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:16.576688 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:16.576763 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:16.577453 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.578220 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.579454 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.579879 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.579994 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:16.588019 (+ 8025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:40353: BlockingRecv error: recv got EOF from 127.12.2.66:40353 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 23:06:16 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:16 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:17.002110 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:16.993574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:53227)
0620 23:06:16.994118 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:16.994133 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:16.994240 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:16.994590 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:16.994593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:16.994994 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:16.995158 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:16.995168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.996442 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.996446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:16.997598 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:16.997613 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.998364 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.998368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:16.998504 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:16.999110 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:16.999127 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:17.001861 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":296,"threads_started":1}
W20260620 23:06:17.002276 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:16.993679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:53227 (local address 127.12.2.126:34119)
0620 23:06:16.993872 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:16.993879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:16.994212 (+ 333us) server_negotiation.cc:407] Connection header received
0620 23:06:16.994344 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:16.994350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:16.994431 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:16.994540 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:16.995304 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.996287 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.997722 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:16.998197 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:16.998296 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:17.002075 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:53227: BlockingRecv error: recv got EOF from 127.12.2.67:53227 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:06:17 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:17 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:17.153991 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:17.145640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:48911)
0620 23:06:17.146034 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:17.146054 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:17.146191 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:17.146625 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:17.146630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:17.147137 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:17.147325 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:17.147335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:17.148379 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:17.148382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:17.149289 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:17.149300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:17.149949 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:17.149955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:17.150068 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:17.150694 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:17.150710 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:17.153752 (+ 3042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":201,"threads_started":1}
W20260620 23:06:17.154258 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:17.145668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:48911 (local address 127.12.2.126:34119)
0620 23:06:17.145995 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:17.146000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:17.146167 (+ 167us) server_negotiation.cc:407] Connection header received
0620 23:06:17.146310 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:17.146314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:17.146439 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:17.146542 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:17.147469 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:17.148269 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:17.149433 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:17.149835 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:17.149937 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:17.153984 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:48911: BlockingRecv error: recv got EOF from 127.12.2.65:48911 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Jun 20 23:06:17 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:17 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:17.597546 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:17.589195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:44773)
0620 23:06:17.589601 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:17.589618 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:17.589734 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:17.590048 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:17.590051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:17.590387 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:17.590569 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:17.590577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:17.591711 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:17.591715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:17.592735 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:17.592748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:17.593557 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:17.593564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:17.593726 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:17.594394 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:17.594412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:17.597264 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":216,"threads_started":1}
W20260620 23:06:17.597723 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:17.589245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:44773 (local address 127.12.2.126:34119)
0620 23:06:17.589463 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:17.589470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:17.589713 (+ 243us) server_negotiation.cc:407] Connection header received
0620 23:06:17.589826 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:17.589830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:17.589899 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:17.589983 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:17.590744 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:17.591586 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:17.592918 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:17.593409 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:17.593528 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:17.597507 (+ 3979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:44773: BlockingRecv error: recv got EOF from 127.12.2.66:44773 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:06:18 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:18 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.67@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:18.010583 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:18.003001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.67:59811)
0620 23:06:18.003402 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:18.003419 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:18.003530 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:18.003791 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:18.003794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:18.004200 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:18.004378 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:18.004387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.005378 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.005383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:18.006278 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:18.006289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.006985 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.006990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:18.007108 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:18.007693 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:18.007710 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:18.010228 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":204,"threads_started":1}
W20260620 23:06:18.010679 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:18.003069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:59811 (local address 127.12.2.126:34119)
0620 23:06:18.003259 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:18.003265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:18.003479 (+ 214us) server_negotiation.cc:407] Connection header received
0620 23:06:18.003609 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:18.003613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:18.003680 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:18.003761 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:18.004515 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.005247 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.006399 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.006807 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.006898 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:18.010467 (+ 3569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.67:59811: BlockingRecv error: recv got EOF from 127.12.2.67:59811 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 23:06:18 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:18 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.65@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:18.162695 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:18.154977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.65:57835)
0620 23:06:18.155421 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:18.155437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:18.155547 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:18.155877 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:18.155882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:18.156381 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:18.156608 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:18.156618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.157584 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.157600 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:18.158485 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:18.158497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.159193 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.159198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:18.159357 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:18.159885 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:18.159901 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:18.162332 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":240,"threads_started":1}
W20260620 23:06:18.162755 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:18.155081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:57835 (local address 127.12.2.126:34119)
0620 23:06:18.155288 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:18.155294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:18.155510 (+ 216us) server_negotiation.cc:407] Connection header received
0620 23:06:18.155626 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:18.155629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:18.155734 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:18.155817 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:18.156740 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.157460 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.158608 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.159062 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.159160 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:18.162537 (+ 3377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.65:57835: BlockingRecv error: recv got EOF from 127.12.2.65:57835 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:06:18 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:06:18 dist-test-slave-7g8g krb5kdc[20520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.2.66@KRBTEST.COM for oryx/127.12.2.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:06:18.606313 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:18.598620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34119 (local address 127.12.2.66:53123)
0620 23:06:18.599130 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:18.599146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:18.599266 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:18.599565 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:18.599570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:18.599976 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:18.600159 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:18.600167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.601132 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.601138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:18.602071 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:18.602083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.602778 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.602784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:18.602997 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:18.603542 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:18.603559 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:18.606072 (+ 2513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.2.126:34119: Server oryx/127.12.2.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":300,"threads_started":1}
W20260620 23:06:18.606426 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:18.598739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:53123 (local address 127.12.2.126:34119)
0620 23:06:18.598923 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:18.598959 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:18.599219 (+ 260us) server_negotiation.cc:407] Connection header received
0620 23:06:18.599375 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:18.599380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:18.599446 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:18.599524 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:18.600286 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.600995 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.602192 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.602648 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.602758 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:18.606230 (+ 3472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.2.66:53123: BlockingRecv error: recv got EOF from 127.12.2.66:53123 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:06:18.858246 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 21107
I20260620 23:06:18.869254 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:18.859144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.12.2.65:32881)
0620 23:06:18.859606 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:18.859612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:18.859638 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:06:18.859733 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:18.859740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:18.859956 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:18.860105 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:18.861371 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.862355 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.863569 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.864126 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.864227 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:18.866546 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:18.866554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:18.866569 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:18.866604 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:18.868791 (+ 2187us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.2.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:06:18.869037 (+ 246us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55374: Request ticket server kudu/127.12.2.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":212,"threads_started":1}
W20260620 23:06:18.869455 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 21107: Not authorized: Client connection negotiation failed: client connection to 127.12.2.65:32881: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.2.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:06:18.869632 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 21107
I20260620 23:06:18.882623 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 21258
I20260620 23:06:18.894130 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:18.883568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.12.2.66:41065)
0620 23:06:18.884123 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:18.884130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:18.884156 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:06:18.884212 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:18.884217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:18.884436 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:18.884611 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:18.885871 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.886873 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.888163 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.888712 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.888838 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:18.891208 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:18.891217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:18.891231 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:18.891275 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:18.893654 (+ 2379us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.2.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:06:18.893884 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46322: Request ticket server kudu/127.12.2.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":249,"threads_started":1}
W20260620 23:06:18.894218 21569 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46322: Request ticket server kudu/127.12.2.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:06:18.894281 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 21258: Not authorized: Client connection negotiation failed: client connection to 127.12.2.66:41065: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.2.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:06:18.894421 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 21258
I20260620 23:06:18.907158 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 21389
I20260620 23:06:18.918255 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:18.908073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.12.2.67:39847)
0620 23:06:18.908602 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:18.908609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:18.908634 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:06:18.908692 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:18.908698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:18.908911 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:18.909062 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:18.910208 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.911343 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.912451 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:18.912976 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:18.913076 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:18.915477 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:18.915485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:18.915512 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:18.915549 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:18.917800 (+ 2251us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.2.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:06:18.918032 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34482: Request ticket server kudu/127.12.2.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":268,"threads_started":1}
W20260620 23:06:18.918352 21570 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34482: Request ticket server kudu/127.12.2.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:06:18.918437 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 21389: Not authorized: Client connection negotiation failed: client connection to 127.12.2.67:39847: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.2.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:06:18.918609 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 21389
I20260620 23:06:18.931290 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 21038
I20260620 23:06:18.932569 21100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:19.043133 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 21038
2026-06-20T23:06:19Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20707 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T23:06:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:06:19Z Disabled control of system clock
I20260620 23:06:19.109750 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:44645
--webserver_interface=127.12.2.126
--webserver_port=0
--builtin_ntp_servers=127.12.2.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:44645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:06:19.424403 21575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:19.424687 21575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:19.424774 21575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:19.431949 21575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:19.432060 21575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:19.432104 21575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:19.432160 21575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:19.432201 21575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:19.441282 21575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:44645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:44645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:19.442612 21575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:19.444315 21575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:19.455839 21581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:19.455868 21583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:19.455895 21580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:19.456642 21575 server_base.cc:1034] running on GCE node
I20260620 23:06:19.457406 21575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:19.458745 21575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:19.459956 21575 hybrid_clock.cc:648] HybridClock initialized: now 1781996779459889 us; error 56 us; skew 500 ppm
I20260620 23:06:19.460253 21575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:06:19.462481 21575 webserver.cc:460] Webserver started at http://127.12.2.126:34845/ using document root <none> and password file <none>
I20260620 23:06:19.463236 21575 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:19.463335 21575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:19.463646 21575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:19.466600 21575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "5445f9e9841940abb999c8d9991e02ab"
format_stamp: "Formatted at 2026-06-20 23:06:19 on dist-test-slave-7g8g"
server_key: "75c1cdcd2c324cb1464639de32569942"
server_key_iv: "85632e69359898ba20c480d8d72c296f"
server_key_version: "encryptionkey@0"
I20260620 23:06:19.467476 21575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "5445f9e9841940abb999c8d9991e02ab"
format_stamp: "Formatted at 2026-06-20 23:06:19 on dist-test-slave-7g8g"
server_key: "75c1cdcd2c324cb1464639de32569942"
server_key_iv: "85632e69359898ba20c480d8d72c296f"
server_key_version: "encryptionkey@0"
I20260620 23:06:19.472594 21575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:06:19.476027 21589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:19.477392 21575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:06:19.477577 21575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "5445f9e9841940abb999c8d9991e02ab"
format_stamp: "Formatted at 2026-06-20 23:06:19 on dist-test-slave-7g8g"
server_key: "75c1cdcd2c324cb1464639de32569942"
server_key_iv: "85632e69359898ba20c480d8d72c296f"
server_key_version: "encryptionkey@0"
I20260620 23:06:19.477779 21575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:19.502068 21575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:19.503090 21575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:19.503386 21575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:19.519773 21575 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:44645
I20260620 23:06:19.519783 21640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:44645 every 8 connection(s)
I20260620 23:06:19.521063 21575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:06:19.524868 21641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:19.528590 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 21575
I20260620 23:06:19.528755 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:06:19.529369 12297 external_mini_cluster.cc:1428] Setting key 5febe7e70618669b6c6c13f4187cb368
I20260620 23:06:19.537336 21641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab: Bootstrap starting.
I20260620 23:06:19.541119 21641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:19.542371 21641 log.cc:824] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:19.545539 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:19.531405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.12.2.126:44645)
0620 23:06:19.532460 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:19.532476 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:19.532531 (+ 55us) server_negotiation.cc:407] Connection header received
0620 23:06:19.535485 (+ 2954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:19.535516 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:19.536176 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:19.536625 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:19.537180 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:19.539163 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:19.540465 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:19.541552 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:19.541762 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:19.542804 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:19.542816 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:19.542834 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:19.543518 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:19.543601 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:19.543636 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:19.543754 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:19.543941 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 23:06:19.544254 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":322,"threads_started":1}
I20260620 23:06:19.545734 21641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab: No bootstrap required, opened a new log
I20260620 23:06:19.553457 21641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5445f9e9841940abb999c8d9991e02ab" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44645 } }
I20260620 23:06:19.553884 21641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:06:19.553998 21641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5445f9e9841940abb999c8d9991e02ab, State: Initialized, Role: FOLLOWER
I20260620 23:06:19.554651 21641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5445f9e9841940abb999c8d9991e02ab" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44645 } }
I20260620 23:06:19.554951 21641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:06:19.555107 21641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:06:19.555271 21641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:19.557461 21641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5445f9e9841940abb999c8d9991e02ab" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44645 } }
I20260620 23:06:19.558005 21641 leader_election.cc:302] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5445f9e9841940abb999c8d9991e02ab; no voters:
I20260620 23:06:19.558595 21641 leader_election.cc:288] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:06:19.558717 21646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:06:19.559964 21646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [term 1 LEADER]: Becoming Leader. State: Replica: 5445f9e9841940abb999c8d9991e02ab, State: Running, Role: LEADER
I20260620 23:06:19.560469 21646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5445f9e9841940abb999c8d9991e02ab" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44645 } }
I20260620 23:06:19.560823 21641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:06:19.562844 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5445f9e9841940abb999c8d9991e02ab. Latest consensus state: current_term: 1 leader_uuid: "5445f9e9841940abb999c8d9991e02ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5445f9e9841940abb999c8d9991e02ab" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44645 } } }
I20260620 23:06:19.563431 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5445f9e9841940abb999c8d9991e02ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5445f9e9841940abb999c8d9991e02ab" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44645 } } }
I20260620 23:06:19.563745 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:19.563818 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:19.565340 21655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:06:19.570058 21655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:06:19.578805 21655 catalog_manager.cc:1347] Generated new cluster ID: a2a5657e30474e50a8745c62dc4ea666
I20260620 23:06:19.578961 21655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:06:19.595019 21655 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:06:19.596230 21655 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:06:19.603945 21655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5445f9e9841940abb999c8d9991e02ab: Generated new TSK 0
I20260620 23:06:19.604738 21655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:06:19.617166 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44645
--builtin_ntp_servers=127.12.2.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 23:06:19.937119 21665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:19.937389 21665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:19.937489 21665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:19.944542 21665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:19.944641 21665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:19.944770 21665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:06:19.953599 21665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:19.954885 21665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:19.956640 21665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:19.969002 21670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:19.969028 21671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:19.969102 21673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:19.969636 21665 server_base.cc:1034] running on GCE node
I20260620 23:06:19.970453 21665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:19.971316 21665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:19.972549 21665 hybrid_clock.cc:648] HybridClock initialized: now 1781996779972479 us; error 56 us; skew 500 ppm
I20260620 23:06:19.972920 21665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:06:19.975033 21665 webserver.cc:460] Webserver started at http://127.12.2.65:46861/ using document root <none> and password file <none>
I20260620 23:06:19.975797 21665 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:19.975884 21665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:19.976150 21665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:19.978909 21665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "a84de621a1ce427d8c1311021c48b5ab"
format_stamp: "Formatted at 2026-06-20 23:06:19 on dist-test-slave-7g8g"
server_key: "42edfb6eeeb892b7ece1f9c7ef2762f3"
server_key_iv: "41518094afd58fde027e2c92c6431959"
server_key_version: "encryptionkey@0"
I20260620 23:06:19.979708 21665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "a84de621a1ce427d8c1311021c48b5ab"
format_stamp: "Formatted at 2026-06-20 23:06:19 on dist-test-slave-7g8g"
server_key: "42edfb6eeeb892b7ece1f9c7ef2762f3"
server_key_iv: "41518094afd58fde027e2c92c6431959"
server_key_version: "encryptionkey@0"
I20260620 23:06:19.984773 21665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:06:19.988235 21679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:19.989675 21665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:06:19.989835 21665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "a84de621a1ce427d8c1311021c48b5ab"
format_stamp: "Formatted at 2026-06-20 23:06:19 on dist-test-slave-7g8g"
server_key: "42edfb6eeeb892b7ece1f9c7ef2762f3"
server_key_iv: "41518094afd58fde027e2c92c6431959"
server_key_version: "encryptionkey@0"
I20260620 23:06:19.990039 21665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:20.010730 21665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:20.011601 21665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:20.011857 21665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:20.012780 21665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:20.014186 21665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:20.014279 21665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:20.014384 21665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:20.014460 21665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:20.043599 21665 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:41335
I20260620 23:06:20.043708 21791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:41335 every 8 connection(s)
I20260620 23:06:20.044857 21665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:06:20.051059 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 21665
I20260620 23:06:20.051232 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:06:20.051837 12297 external_mini_cluster.cc:1428] Setting key 68c7d144c492b89dc6cbd3edc50d48d9
I20260620 23:06:20.054194 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44645
--builtin_ntp_servers=127.12.2.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:06:20.056660 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:20.047238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:34563 (local address 127.12.2.126:44645)
0620 23:06:20.047687 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:20.047692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:20.048852 (+ 1160us) server_negotiation.cc:407] Connection header received
0620 23:06:20.050543 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:20.050548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:20.050629 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:20.050721 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:20.051846 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.052584 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.054043 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.054464 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.054576 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:20.055834 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:20.055839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:20.055842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:20.055858 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:20.055893 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:20.055904 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:20.055973 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:20.056174 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 23:06:20.056262 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260620 23:06:20.057570 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:20.047856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:44645 (local address 127.12.2.65:34563)
0620 23:06:20.048692 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:20.048731 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:20.050284 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:20.050983 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:20.050999 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:20.051092 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:20.051696 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:20.051712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.052730 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.052735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:20.053914 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:20.053925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.054670 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.054677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:20.054895 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:20.055544 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:06:20.055546 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:20.055584 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:06:20.055659 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:20.055984 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:20.056003 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:20.056009 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:06:20.056154 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 23:06:20.056534 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":230,"threads_started":1}
I20260620 23:06:20.059094 21792 heartbeater.cc:347] Connected to a master server at 127.12.2.126:44645
I20260620 23:06:20.059362 21792 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:20.060094 21792 heartbeater.cc:511] Master 127.12.2.126:44645 requested a full tablet report, sending...
I20260620 23:06:20.062002 21606 ts_manager.cc:194] Registered new tserver with Master: a84de621a1ce427d8c1311021c48b5ab (127.12.2.65:41335)
I20260620 23:06:20.063524 21606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.65:34563
W20260620 23:06:20.364852 21796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:20.365144 21796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:20.365267 21796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:20.372363 21796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:20.372473 21796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:20.372608 21796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:06:20.381541 21796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:20.382840 21796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:20.384603 21796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:20.397073 21804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:20.397104 21802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:20.397265 21801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:20.397931 21796 server_base.cc:1034] running on GCE node
I20260620 23:06:20.398507 21796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:20.399507 21796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:20.400691 21796 hybrid_clock.cc:648] HybridClock initialized: now 1781996780400638 us; error 42 us; skew 500 ppm
I20260620 23:06:20.400987 21796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:06:20.403112 21796 webserver.cc:460] Webserver started at http://127.12.2.66:45155/ using document root <none> and password file <none>
I20260620 23:06:20.403769 21796 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:20.403864 21796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:20.404161 21796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:20.406831 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "6496280c890749aca8461c14275963e8"
format_stamp: "Formatted at 2026-06-20 23:06:20 on dist-test-slave-7g8g"
server_key: "e1136e38d5151fa2b80b41251515f640"
server_key_iv: "cd2d30eea2df583e07ac055c5c359242"
server_key_version: "encryptionkey@0"
I20260620 23:06:20.407639 21796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "6496280c890749aca8461c14275963e8"
format_stamp: "Formatted at 2026-06-20 23:06:20 on dist-test-slave-7g8g"
server_key: "e1136e38d5151fa2b80b41251515f640"
server_key_iv: "cd2d30eea2df583e07ac055c5c359242"
server_key_version: "encryptionkey@0"
I20260620 23:06:20.412413 21796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:06:20.415656 21810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:20.416905 21796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:06:20.417055 21796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "6496280c890749aca8461c14275963e8"
format_stamp: "Formatted at 2026-06-20 23:06:20 on dist-test-slave-7g8g"
server_key: "e1136e38d5151fa2b80b41251515f640"
server_key_iv: "cd2d30eea2df583e07ac055c5c359242"
server_key_version: "encryptionkey@0"
I20260620 23:06:20.417245 21796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:20.440217 21796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:20.441064 21796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:20.441325 21796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:20.442302 21796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:20.443884 21796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:20.443980 21796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:20.444087 21796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:20.444161 21796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:20.473803 21796 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:39409
I20260620 23:06:20.473869 21922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:39409 every 8 connection(s)
I20260620 23:06:20.475090 21796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:06:20.485522 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 21796
I20260620 23:06:20.485666 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:06:20.486267 12297 external_mini_cluster.cc:1428] Setting key cb394412ff3f358892216b0f3f3fdc6a
I20260620 23:06:20.486907 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:20.477454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:35863 (local address 127.12.2.126:44645)
0620 23:06:20.477614 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:20.477629 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:20.479020 (+ 1391us) server_negotiation.cc:407] Connection header received
0620 23:06:20.480736 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:20.480740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:20.480808 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:20.480884 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:20.482047 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.482777 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.484308 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.484702 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.484802 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:20.486171 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:20.486177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:20.486180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:20.486198 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:20.486235 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:20.486248 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:20.486318 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:20.486570 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 23:06:20.486648 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:06:20.488194 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:20.477956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:44645 (local address 127.12.2.66:35863)
0620 23:06:20.478836 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:20.478871 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:20.480477 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:20.481113 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:20.481121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:20.481218 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:20.481828 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:20.481845 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.482912 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.482916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:20.484170 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:20.484181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.484893 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.484898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:20.485135 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:20.485817 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:06:20.485821 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:20.485873 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:06:20.485979 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:20.486346 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:20.486357 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:20.486365 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:06:20.486560 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 23:06:20.486957 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":268,"threads_started":1}
I20260620 23:06:20.489277 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44645
--builtin_ntp_servers=127.12.2.84:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:06:20.489534 21923 heartbeater.cc:347] Connected to a master server at 127.12.2.126:44645
I20260620 23:06:20.489804 21923 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:20.490509 21923 heartbeater.cc:511] Master 127.12.2.126:44645 requested a full tablet report, sending...
I20260620 23:06:20.491854 21606 ts_manager.cc:194] Registered new tserver with Master: 6496280c890749aca8461c14275963e8 (127.12.2.66:39409)
I20260620 23:06:20.492777 21606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.66:35863
W20260620 23:06:20.802141 21926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:20.802402 21926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:20.802521 21926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:20.809661 21926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:20.809767 21926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:20.809908 21926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:06:20.818784 21926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:20.819990 21926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:20.821727 21926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:20.833465 21931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:20.833468 21932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:20.833664 21934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:20.834316 21926 server_base.cc:1034] running on GCE node
I20260620 23:06:20.834853 21926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:20.835739 21926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:20.836920 21926 hybrid_clock.cc:648] HybridClock initialized: now 1781996780836860 us; error 52 us; skew 500 ppm
I20260620 23:06:20.837203 21926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:06:20.839270 21926 webserver.cc:460] Webserver started at http://127.12.2.67:34957/ using document root <none> and password file <none>
I20260620 23:06:20.839906 21926 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:20.839993 21926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:20.840293 21926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:20.842917 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "005984ea70a6431090722708967182d6"
format_stamp: "Formatted at 2026-06-20 23:06:20 on dist-test-slave-7g8g"
server_key: "62fec67ce80a00db86619ceb1d4be687"
server_key_iv: "a17cb0a8274908823bb6bfee627020e0"
server_key_version: "encryptionkey@0"
I20260620 23:06:20.843709 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "005984ea70a6431090722708967182d6"
format_stamp: "Formatted at 2026-06-20 23:06:20 on dist-test-slave-7g8g"
server_key: "62fec67ce80a00db86619ceb1d4be687"
server_key_iv: "a17cb0a8274908823bb6bfee627020e0"
server_key_version: "encryptionkey@0"
I20260620 23:06:20.848465 21926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:06:20.851737 21940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:20.853067 21926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:06:20.853202 21926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "005984ea70a6431090722708967182d6"
format_stamp: "Formatted at 2026-06-20 23:06:20 on dist-test-slave-7g8g"
server_key: "62fec67ce80a00db86619ceb1d4be687"
server_key_iv: "a17cb0a8274908823bb6bfee627020e0"
server_key_version: "encryptionkey@0"
I20260620 23:06:20.853389 21926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:20.885200 21926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:20.886061 21926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:20.886353 21926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:20.887346 21926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:20.888739 21926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:20.888823 21926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:20.888903 21926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:20.888950 21926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:20.917814 21926 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:40691
I20260620 23:06:20.917870 22052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:40691 every 8 connection(s)
I20260620 23:06:20.919063 21926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:06:20.921617 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 21926
I20260620 23:06:20.921774 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:06:20.922180 12297 external_mini_cluster.cc:1428] Setting key 48d4ec56c2202af1ac4bb6c13761ccad
I20260620 23:06:20.930733 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:20.921483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:40449 (local address 127.12.2.126:44645)
0620 23:06:20.921706 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:20.921712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:20.923265 (+ 1553us) server_negotiation.cc:407] Connection header received
0620 23:06:20.925260 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:20.925263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:20.925328 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:20.925404 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:20.926358 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.927110 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.928578 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.928946 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.929041 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:20.930157 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:20.930161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:20.930164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:20.930180 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:20.930209 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:20.930220 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:20.930283 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:20.930432 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:06:20.930504 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:06:20.931668 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:20.922127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:44645 (local address 127.12.2.67:40449)
0620 23:06:20.923122 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:20.923166 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:20.924984 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:20.925575 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:20.925583 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:20.925669 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:20.926219 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:20.926235 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.927225 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.927229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:20.928457 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:20.928468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.929118 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.929122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:20.929309 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:20.929880 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:06:20.929882 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:20.929919 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:06:20.929989 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:20.930286 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:20.930291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:20.930297 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:06:20.930419 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:06:20.930695 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":265,"threads_started":1}
I20260620 23:06:20.932844 22053 heartbeater.cc:347] Connected to a master server at 127.12.2.126:44645
I20260620 23:06:20.933112 22053 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:20.933954 22053 heartbeater.cc:511] Master 127.12.2.126:44645 requested a full tablet report, sending...
I20260620 23:06:20.935369 21606 ts_manager.cc:194] Registered new tserver with Master: 005984ea70a6431090722708967182d6 (127.12.2.67:40691)
I20260620 23:06:20.936244 21606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.67:40449
I20260620 23:06:20.937309 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:06:20.943167 21794 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:20.941683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.12.2.126:44645)
0620 23:06:20.941911 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:20.941918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:20.942293 (+ 375us) server_negotiation.cc:407] Connection header received
0620 23:06:20.942428 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:20.942432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:20.942503 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:20.942576 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:20.942923 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35212: BlockingRecv error: recv got EOF from 127.0.0.1:35212 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:06:20.943801 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 21665
I20260620 23:06:20.951202 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:20.944675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.12.2.65:41335)
0620 23:06:20.945162 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:20.945168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:20.945192 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:06:20.945242 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:20.945248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:20.945438 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:20.945554 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:20.945865 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.946839 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.948114 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:20.948662 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:20.948758 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:20.949792 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:20.949800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:20.949812 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:20.950399 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:20.950443 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:20.950461 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:20.950549 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:20.950708 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 23:06:20.950812 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":222,"threads_started":1}
I20260620 23:06:20.952359 21787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:21.032269 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 21665
I20260620 23:06:21.045239 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 21796
I20260620 23:06:21.053097 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:21.046177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.12.2.66:39409)
0620 23:06:21.046681 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:21.046689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:21.046715 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:06:21.046796 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:21.046803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:21.047079 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:21.047203 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:21.047541 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:21.048615 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:21.049969 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:21.050534 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:21.050652 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:21.051685 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:21.051694 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:21.051708 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:21.052302 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:21.052350 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:21.052369 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:21.052464 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:21.052600 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 23:06:21.052707 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":247,"threads_started":1}
I20260620 23:06:21.054294 21918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:21.134392 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 21796
I20260620 23:06:21.146867 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 21926
I20260620 23:06:21.154726 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:21.147893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.12.2.67:40691)
0620 23:06:21.148499 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:21.148506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:21.148533 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:06:21.148588 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:21.148594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:21.148804 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:21.148950 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:21.149381 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:21.150420 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:21.151704 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:21.152252 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:21.152361 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:21.153358 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:21.153365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:21.153379 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:21.153939 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:21.154021 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:21.154040 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:21.154133 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:21.154281 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 23:06:21.154411 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":321,"threads_started":1}
I20260620 23:06:21.156144 22048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:21.235778 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 21926
I20260620 23:06:21.247771 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 21575
I20260620 23:06:21.248854 21636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:21.351295 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 21575
2026-06-20T23:06:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2303 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T23:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:06:21Z Disabled control of system clock
I20260620 23:06:21.412585 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:34971
--webserver_interface=127.12.2.126
--webserver_port=0
--builtin_ntp_servers=127.12.2.84:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:34971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:06:21.718580 22072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:21.718858 22072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:21.719004 22072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:21.725998 22072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:21.726104 22072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:21.726140 22072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:21.726197 22072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:21.726239 22072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:21.735224 22072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:34971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:34971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:21.736406 22072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:21.738116 22072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:21.749151 22077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:21.749118 22078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:21.749186 22080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:21.749727 22072 server_base.cc:1034] running on GCE node
I20260620 23:06:21.750458 22072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:21.751830 22072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:21.753067 22072 hybrid_clock.cc:648] HybridClock initialized: now 1781996781753033 us; error 59 us; skew 500 ppm
I20260620 23:06:21.753403 22072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:06:21.760057 22072 webserver.cc:460] Webserver started at http://127.12.2.126:45081/ using document root <none> and password file <none>
I20260620 23:06:21.760779 22072 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:21.760871 22072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:21.761181 22072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:21.764091 22072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "9ac0d1beb75a42fab8cc6671dac0ab68"
format_stamp: "Formatted at 2026-06-20 23:06:21 on dist-test-slave-7g8g"
server_key: "cceea1bb805922e73e804c1b3f5525df"
server_key_iv: "e68a9fe8734538049a3a54a3c0fd9885"
server_key_version: "encryptionkey@0"
I20260620 23:06:21.764844 22072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "9ac0d1beb75a42fab8cc6671dac0ab68"
format_stamp: "Formatted at 2026-06-20 23:06:21 on dist-test-slave-7g8g"
server_key: "cceea1bb805922e73e804c1b3f5525df"
server_key_iv: "e68a9fe8734538049a3a54a3c0fd9885"
server_key_version: "encryptionkey@0"
I20260620 23:06:21.769829 22072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:06:21.773149 22086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:21.774461 22072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:06:21.774616 22072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "9ac0d1beb75a42fab8cc6671dac0ab68"
format_stamp: "Formatted at 2026-06-20 23:06:21 on dist-test-slave-7g8g"
server_key: "cceea1bb805922e73e804c1b3f5525df"
server_key_iv: "e68a9fe8734538049a3a54a3c0fd9885"
server_key_version: "encryptionkey@0"
I20260620 23:06:21.774802 22072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:21.780349 22072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:21.781088 22072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:21.781352 22072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:21.798043 22072 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:34971
I20260620 23:06:21.798110 22137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:34971 every 8 connection(s)
I20260620 23:06:21.799365 22072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:06:21.800189 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 22072
I20260620 23:06:21.800400 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:06:21.801020 12297 external_mini_cluster.cc:1428] Setting key e6c48b91aa7308cd14aa6631157f0ff5
I20260620 23:06:21.803555 22138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:21.810232 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:21.803058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.12.2.126:34971)
0620 23:06:21.804104 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:21.804119 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:21.804171 (+ 52us) server_negotiation.cc:407] Connection header received
0620 23:06:21.806203 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:21.806229 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:21.806772 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:21.807245 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:21.808256 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:21.808264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:21.808280 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:21.808295 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:21.808387 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:21.808412 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:21.808625 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:21.808747 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 23:06:21.809031 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":297,"threads_started":1}
I20260620 23:06:21.816433 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68: Bootstrap starting.
I20260620 23:06:21.819285 22138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:21.820214 22138 log.cc:824] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:21.823381 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68: No bootstrap required, opened a new log
I20260620 23:06:21.830602 22138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac0d1beb75a42fab8cc6671dac0ab68" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34971 } }
I20260620 23:06:21.830987 22138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:06:21.831073 22138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac0d1beb75a42fab8cc6671dac0ab68, State: Initialized, Role: FOLLOWER
I20260620 23:06:21.831607 22138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac0d1beb75a42fab8cc6671dac0ab68" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34971 } }
I20260620 23:06:21.831810 22138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:06:21.831900 22138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:06:21.832027 22138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:21.834297 22138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac0d1beb75a42fab8cc6671dac0ab68" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34971 } }
I20260620 23:06:21.834816 22138 leader_election.cc:302] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ac0d1beb75a42fab8cc6671dac0ab68; no voters:
I20260620 23:06:21.835413 22138 leader_election.cc:288] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:06:21.835572 22143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:06:21.836210 22143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [term 1 LEADER]: Becoming Leader. State: Replica: 9ac0d1beb75a42fab8cc6671dac0ab68, State: Running, Role: LEADER
I20260620 23:06:21.836753 22143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac0d1beb75a42fab8cc6671dac0ab68" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34971 } }
I20260620 23:06:21.837376 22138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:06:21.839921 22144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ac0d1beb75a42fab8cc6671dac0ab68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac0d1beb75a42fab8cc6671dac0ab68" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34971 } } }
I20260620 23:06:21.840231 22144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:21.840961 22145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac0d1beb75a42fab8cc6671dac0ab68. Latest consensus state: current_term: 1 leader_uuid: "9ac0d1beb75a42fab8cc6671dac0ab68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac0d1beb75a42fab8cc6671dac0ab68" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 34971 } } }
I20260620 23:06:21.841336 22145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:21.842376 22152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:06:21.846822 22152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:06:21.855139 22152 catalog_manager.cc:1347] Generated new cluster ID: b4346537fe7d4be991b215683e8c7cde
I20260620 23:06:21.855257 22152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:06:21.892930 22152 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:06:21.893424 22152 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:06:21.903928 22152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ac0d1beb75a42fab8cc6671dac0ab68: Generated new TSK 0
I20260620 23:06:21.904939 22152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:06:21.920878 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34971
--builtin_ntp_servers=127.12.2.84:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:06:22.232547 22162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:22.232820 22162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:22.232928 22162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:22.240175 22162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:22.240274 22162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:22.240408 22162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:06:22.249454 22162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:22.250720 22162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:22.252486 22162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:22.264988 22170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:22.265126 22167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:22.265110 22168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:22.266045 22162 server_base.cc:1034] running on GCE node
I20260620 23:06:22.266656 22162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:22.267627 22162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:22.268863 22162 hybrid_clock.cc:648] HybridClock initialized: now 1781996782268808 us; error 45 us; skew 500 ppm
I20260620 23:06:22.269208 22162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:06:22.271411 22162 webserver.cc:460] Webserver started at http://127.12.2.65:43787/ using document root <none> and password file <none>
I20260620 23:06:22.272104 22162 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:22.272221 22162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:22.272571 22162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:22.275419 22162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "434f7919b8b943c5a2c35a52fb2b6eaa"
format_stamp: "Formatted at 2026-06-20 23:06:22 on dist-test-slave-7g8g"
server_key: "801effc9f58faeccd3e0e0eda0aa0494"
server_key_iv: "ea1ce6066582dc9e3d6c91d83040e2f3"
server_key_version: "encryptionkey@0"
I20260620 23:06:22.276160 22162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "434f7919b8b943c5a2c35a52fb2b6eaa"
format_stamp: "Formatted at 2026-06-20 23:06:22 on dist-test-slave-7g8g"
server_key: "801effc9f58faeccd3e0e0eda0aa0494"
server_key_iv: "ea1ce6066582dc9e3d6c91d83040e2f3"
server_key_version: "encryptionkey@0"
I20260620 23:06:22.281350 22162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:06:22.284684 22176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:22.286057 22162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:06:22.286201 22162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "434f7919b8b943c5a2c35a52fb2b6eaa"
format_stamp: "Formatted at 2026-06-20 23:06:22 on dist-test-slave-7g8g"
server_key: "801effc9f58faeccd3e0e0eda0aa0494"
server_key_iv: "ea1ce6066582dc9e3d6c91d83040e2f3"
server_key_version: "encryptionkey@0"
I20260620 23:06:22.286396 22162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:22.291652 22162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:22.292375 22162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:22.292619 22162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:22.293444 22162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:22.294765 22162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:22.294848 22162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:22.294965 22162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:22.295038 22162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 23:06:22.323042 22162 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:35881
I20260620 23:06:22.323087 22288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:35881 every 8 connection(s)
I20260620 23:06:22.324321 22162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:06:22.332103 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:22.326733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:39533 (local address 127.12.2.126:34971)
0620 23:06:22.327242 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:22.327247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:22.328375 (+ 1128us) server_negotiation.cc:407] Connection header received
0620 23:06:22.330013 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:22.330018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:22.330077 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:22.330167 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:22.331419 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:22.331424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:22.331429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:22.331448 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:22.331486 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:22.331498 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:22.331596 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:22.331766 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 23:06:22.331827 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":296,"threads_started":1}
I20260620 23:06:22.332957 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:22.327331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34971 (local address 127.12.2.65:39533)
0620 23:06:22.328226 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:22.328268 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:22.329752 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:22.330402 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:22.330410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:22.330510 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:22.331140 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:06:22.331142 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:22.331178 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:06:22.331259 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:22.331580 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:22.331590 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:22.331597 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:06:22.331834 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 23:06:22.332075 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":258,"threads_started":1}
I20260620 23:06:22.333581 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 22162
I20260620 23:06:22.333788 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:06:22.334298 22289 heartbeater.cc:347] Connected to a master server at 127.12.2.126:34971
I20260620 23:06:22.334420 12297 external_mini_cluster.cc:1428] Setting key aa34d5e3dfa584e6f9cacac78a802ebe
I20260620 23:06:22.334609 22289 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:22.335253 22289 heartbeater.cc:511] Master 127.12.2.126:34971 requested a full tablet report, sending...
I20260620 23:06:22.337112 22103 ts_manager.cc:194] Registered new tserver with Master: 434f7919b8b943c5a2c35a52fb2b6eaa (127.12.2.65:35881)
I20260620 23:06:22.337127 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34971
--builtin_ntp_servers=127.12.2.84:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:06:22.639050 22293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:22.639292 22293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:22.639406 22293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:22.646258 22293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:22.646345 22293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:22.646476 22293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:06:22.655143 22293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:22.656314 22293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:22.657999 22293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:22.670061 22298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:22.670028 22301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:22.670104 22299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:22.670869 22293 server_base.cc:1034] running on GCE node
I20260620 23:06:22.671429 22293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:22.672267 22293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:22.673450 22293 hybrid_clock.cc:648] HybridClock initialized: now 1781996782673412 us; error 30 us; skew 500 ppm
I20260620 23:06:22.673717 22293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:06:22.675740 22293 webserver.cc:460] Webserver started at http://127.12.2.66:41573/ using document root <none> and password file <none>
I20260620 23:06:22.676366 22293 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:22.676447 22293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:22.676703 22293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:22.681022 22293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "ff0d5d9c1cad4fa98069866c20a1eb82"
format_stamp: "Formatted at 2026-06-20 23:06:22 on dist-test-slave-7g8g"
server_key: "5466a9d57aed777889daba01e680f014"
server_key_iv: "a955a42fd7abe9a2a5b13f312cf03e53"
server_key_version: "encryptionkey@0"
I20260620 23:06:22.681764 22293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "ff0d5d9c1cad4fa98069866c20a1eb82"
format_stamp: "Formatted at 2026-06-20 23:06:22 on dist-test-slave-7g8g"
server_key: "5466a9d57aed777889daba01e680f014"
server_key_iv: "a955a42fd7abe9a2a5b13f312cf03e53"
server_key_version: "encryptionkey@0"
I20260620 23:06:22.686721 22293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:06:22.690105 22307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:22.691557 22293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:06:22.691694 22293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "ff0d5d9c1cad4fa98069866c20a1eb82"
format_stamp: "Formatted at 2026-06-20 23:06:22 on dist-test-slave-7g8g"
server_key: "5466a9d57aed777889daba01e680f014"
server_key_iv: "a955a42fd7abe9a2a5b13f312cf03e53"
server_key_version: "encryptionkey@0"
I20260620 23:06:22.691890 22293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:22.697217 22293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:22.697937 22293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:22.698168 22293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:22.699038 22293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:22.700344 22293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:22.700426 22293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:22.700531 22293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:22.700600 22293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:22.729790 22293 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:34121
I20260620 23:06:22.729836 22419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:34121 every 8 connection(s)
I20260620 23:06:22.731068 22293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:06:22.737169 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 22293
I20260620 23:06:22.737320 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:06:22.737797 12297 external_mini_cluster.cc:1428] Setting key 7e4c83ff50c75d52a3f0902bccaada3e
I20260620 23:06:22.738860 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:22.733521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:39205 (local address 127.12.2.126:34971)
0620 23:06:22.733687 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:22.733691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:22.735072 (+ 1381us) server_negotiation.cc:407] Connection header received
0620 23:06:22.736688 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:22.736694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:22.736768 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:22.736850 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:22.738148 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:22.738154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:22.738160 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:22.738178 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:22.738207 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:22.738218 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:22.738314 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:22.738499 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 23:06:22.738571 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:06:22.739710 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:22.734012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34971 (local address 127.12.2.66:39205)
0620 23:06:22.734894 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:22.734955 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:22.736431 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:22.737089 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:22.737097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:22.737186 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:22.737822 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:06:22.737825 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:22.737875 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:06:22.737982 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:22.738297 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:22.738306 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:22.738314 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:06:22.738560 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 23:06:22.738799 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":261,"threads_started":1}
I20260620 23:06:22.740554 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34971
--builtin_ntp_servers=127.12.2.84:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 23:06:22.740963 22420 heartbeater.cc:347] Connected to a master server at 127.12.2.126:34971
I20260620 23:06:22.741286 22420 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:22.741940 22420 heartbeater.cc:511] Master 127.12.2.126:34971 requested a full tablet report, sending...
I20260620 23:06:22.743376 22103 ts_manager.cc:194] Registered new tserver with Master: ff0d5d9c1cad4fa98069866c20a1eb82 (127.12.2.66:34121)
W20260620 23:06:23.055080 22423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:23.055343 22423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:23.055449 22423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:23.062702 22423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:23.062805 22423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:23.062916 22423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:06:23.072084 22423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:34971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:23.073365 22423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:23.075110 22423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:23.087538 22431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:23.087570 22429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:23.087760 22428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:23.088714 22423 server_base.cc:1034] running on GCE node
I20260620 23:06:23.089361 22423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:23.090355 22423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:23.091555 22423 hybrid_clock.cc:648] HybridClock initialized: now 1781996783091503 us; error 41 us; skew 500 ppm
I20260620 23:06:23.091856 22423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:06:23.093953 22423 webserver.cc:460] Webserver started at http://127.12.2.67:37569/ using document root <none> and password file <none>
I20260620 23:06:23.094606 22423 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:23.094704 22423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:23.095039 22423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:23.097885 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "55329063b20c4686a0c9ad1dbc3df440"
format_stamp: "Formatted at 2026-06-20 23:06:23 on dist-test-slave-7g8g"
server_key: "bbf3b48fd5809559b93577530f8c10f7"
server_key_iv: "e24a19b1a3667adae1fd0f8b941d1cf4"
server_key_version: "encryptionkey@0"
I20260620 23:06:23.098630 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "55329063b20c4686a0c9ad1dbc3df440"
format_stamp: "Formatted at 2026-06-20 23:06:23 on dist-test-slave-7g8g"
server_key: "bbf3b48fd5809559b93577530f8c10f7"
server_key_iv: "e24a19b1a3667adae1fd0f8b941d1cf4"
server_key_version: "encryptionkey@0"
I20260620 23:06:23.103576 22423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 23:06:23.106875 22437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:23.108295 22423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:06:23.108445 22423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "55329063b20c4686a0c9ad1dbc3df440"
format_stamp: "Formatted at 2026-06-20 23:06:23 on dist-test-slave-7g8g"
server_key: "bbf3b48fd5809559b93577530f8c10f7"
server_key_iv: "e24a19b1a3667adae1fd0f8b941d1cf4"
server_key_version: "encryptionkey@0"
I20260620 23:06:23.108657 22423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:23.113845 22423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:23.114588 22423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:23.114823 22423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:23.115702 22423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:23.117028 22423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:23.117120 22423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:23.117221 22423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:23.117293 22423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:23.146610 22423 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:45947
I20260620 23:06:23.146672 22549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:45947 every 8 connection(s)
I20260620 23:06:23.147897 22423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:06:23.152096 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 22423
I20260620 23:06:23.152258 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:06:23.152652 12297 external_mini_cluster.cc:1428] Setting key 91d99ea5ffaabf73931f5d7925a63add
I20260620 23:06:23.155969 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:23.150350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:44295 (local address 127.12.2.126:34971)
0620 23:06:23.150549 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:23.150555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:23.151993 (+ 1438us) server_negotiation.cc:407] Connection header received
0620 23:06:23.153742 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:23.153747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:23.153809 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:23.153910 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:23.155348 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:23.155353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:23.155358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:23.155377 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:23.155415 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:23.155430 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:23.155562 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:23.155729 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 23:06:23.155774 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:06:23.156908 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:23.150907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:34971 (local address 127.12.2.67:44295)
0620 23:06:23.151838 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:23.151873 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:23.153500 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:23.154182 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:23.154190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:23.154286 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:23.155046 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:06:23.155049 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:23.155092 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:06:23.155183 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:23.155525 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:23.155534 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:23.155541 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:06:23.155801 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 23:06:23.155995 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":271,"threads_started":1}
I20260620 23:06:23.158121 22550 heartbeater.cc:347] Connected to a master server at 127.12.2.126:34971
I20260620 23:06:23.158353 22550 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:23.158859 22550 heartbeater.cc:511] Master 127.12.2.126:34971 requested a full tablet report, sending...
I20260620 23:06:23.160166 22103 ts_manager.cc:194] Registered new tserver with Master: 55329063b20c4686a0c9ad1dbc3df440 (127.12.2.67:45947)
I20260620 23:06:23.167680 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:06:23.173656 22291 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:23.172223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.12.2.126:34971)
0620 23:06:23.172380 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:23.172384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:23.172937 (+ 553us) server_negotiation.cc:407] Connection header received
0620 23:06:23.173078 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:23.173082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:23.173130 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:23.173205 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:23.173475 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37512: BlockingRecv error: recv got EOF from 127.0.0.1:37512 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:06:23.174342 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 22162
I20260620 23:06:23.177714 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:23.175394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.12.2.65:35881)
0620 23:06:23.175841 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:23.175848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:23.175872 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:06:23.175924 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:23.175929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:23.176159 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:23.176275 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:23.177125 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:23.177132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:23.177142 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:23.177154 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:23.177197 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:23.177214 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:23.177311 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:23.177357 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 23:06:23.177440 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":200,"threads_started":1}
I20260620 23:06:23.178881 22284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:23.258112 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 22162
I20260620 23:06:23.270773 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 22293
I20260620 23:06:23.273975 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:23.271664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.12.2.66:34121)
0620 23:06:23.272187 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:23.272194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:23.272221 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:06:23.272283 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:23.272289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:23.272507 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:23.272632 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:23.273427 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:23.273433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:23.273445 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:23.273462 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:23.273502 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:23.273516 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:23.273617 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:23.273674 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 23:06:23.273750 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":235,"threads_started":1}
I20260620 23:06:23.275239 22415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:23.368417 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 22293
I20260620 23:06:23.382687 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 22423
I20260620 23:06:23.386214 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:23.383524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.12.2.67:45947)
0620 23:06:23.384061 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:23.384068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:23.384095 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:06:23.384152 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:23.384159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:23.384364 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:23.384499 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:23.385512 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:23.385547 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:23.385560 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:23.385573 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:23.385614 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:23.385628 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:23.385774 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:23.385843 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 23:06:23.385940 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":240,"threads_started":1}
I20260620 23:06:23.387413 22545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:23.468714 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 22423
I20260620 23:06:23.480839 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 22072
I20260620 23:06:23.481925 22133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:23.582988 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 22072
2026-06-20T23:06:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2231 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:06:23 dist-test-slave-7g8g krb5kdc[22568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:06:23 dist-test-slave-7g8g krb5kdc[22568](info): set up 2 sockets
Jun 20 23:06:23 dist-test-slave-7g8g krb5kdc[22568](info): commencing operation
krb5kdc: starting...
W20260620 23:06:24.583493 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:06:24 dist-test-slave-7g8g krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:06:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:06:24.790493 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:40965
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:40965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:25.094007 22581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:25.094276 22581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:25.094400 22581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:25.101502 22581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:25.101608 22581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:25.101681 22581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:25.101752 22581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:25.101809 22581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:25.111405 22581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:40965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:40965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:25.112670 22581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:25.113899 22581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:25.124605 22586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:25.124541 22587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:25.124584 22589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:25.125732 22581 server_base.cc:1034] running on GCE node
I20260620 23:06:25.126426 22581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:25.127794 22581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:25.128994 22581 hybrid_clock.cc:648] HybridClock initialized: now 1781996785128946 us; error 38 us; skew 500 ppm
I20260620 23:06:25.129321 22581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:06:25 dist-test-slave-7g8g krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:25.132230 22581 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:06:25.133643 22581 webserver.cc:460] Webserver started at http://127.12.2.126:34871/ using document root <none> and password file <none>
I20260620 23:06:25.134303 22581 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:25.134414 22581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:25.134686 22581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:25.137588 22581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "b18c7fd46b7e4ed686ea29edc78c5242"
format_stamp: "Formatted at 2026-06-20 23:06:25 on dist-test-slave-7g8g"
server_key: "f3aafbeaa2ee47bffb8ea452f54c2edc"
server_key_iv: "b65cf8e46d9a4a3fdc81b992997a997d"
server_key_version: "encryptionkey@0"
I20260620 23:06:25.138324 22581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "b18c7fd46b7e4ed686ea29edc78c5242"
format_stamp: "Formatted at 2026-06-20 23:06:25 on dist-test-slave-7g8g"
server_key: "f3aafbeaa2ee47bffb8ea452f54c2edc"
server_key_iv: "b65cf8e46d9a4a3fdc81b992997a997d"
server_key_version: "encryptionkey@0"
I20260620 23:06:25.143270 22581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:06:25.146564 22596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:25.147859 22581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:06:25.148008 22581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "b18c7fd46b7e4ed686ea29edc78c5242"
format_stamp: "Formatted at 2026-06-20 23:06:25 on dist-test-slave-7g8g"
server_key: "f3aafbeaa2ee47bffb8ea452f54c2edc"
server_key_iv: "b65cf8e46d9a4a3fdc81b992997a997d"
server_key_version: "encryptionkey@0"
I20260620 23:06:25.148185 22581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:25.208559 22581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:25.212417 22581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:25.212689 22581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:25.229043 22581 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:40965
I20260620 23:06:25.229082 22647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:40965 every 8 connection(s)
I20260620 23:06:25.230338 22581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:06:25.233906 22648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:25.239758 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 22581
I20260620 23:06:25.240012 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:06:25.240635 12297 external_mini_cluster.cc:1428] Setting key d980d1c088c46d95d1a48e78df6604f6
I20260620 23:06:25.244290 22648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242: Bootstrap starting.
I20260620 23:06:25.247977 22648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:25.249264 22648 log.cc:824] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:25.252298 22648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242: No bootstrap required, opened a new log
Jun 20 23:06:25 dist-test-slave-7g8g krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:06:25.261818 22648 raft_consensus.cc:348] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18c7fd46b7e4ed686ea29edc78c5242" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40965 } }
I20260620 23:06:25.262259 22648 raft_consensus.cc:374] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:06:25.262358 22648 raft_consensus.cc:729] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18c7fd46b7e4ed686ea29edc78c5242, State: Initialized, Role: FOLLOWER
I20260620 23:06:25.263029 22648 consensus_queue.cc:260] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18c7fd46b7e4ed686ea29edc78c5242" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40965 } }
I20260620 23:06:25.263260 22648 raft_consensus.cc:388] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:06:25.263437 22648 raft_consensus.cc:482] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:06:25.263623 22648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:25.265287 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:25.242771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.12.2.126:40965)
0620 23:06:25.243567 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:25.243582 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:25.243632 (+ 50us) server_negotiation.cc:407] Connection header received
0620 23:06:25.245844 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:25.245879 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:25.246417 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:25.246859 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:25.248148 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:25.250014 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:25.251540 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:25.252520 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:25.252686 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:25.256853 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:25.256862 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:25.256881 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:25.256932 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:25.260726 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:25.261371 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:25.261378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:25.261385 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:25.261585 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:25.261930 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:25.261935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:25.261937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:25.262616 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:25.262899 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:25.263500 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 23:06:25.263929 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":336,"threads_started":1}
I20260620 23:06:25.266381 22648 raft_consensus.cc:504] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18c7fd46b7e4ed686ea29edc78c5242" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40965 } }
I20260620 23:06:25.266847 22648 leader_election.cc:302] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b18c7fd46b7e4ed686ea29edc78c5242; no voters:
I20260620 23:06:25.267469 22648 leader_election.cc:288] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:06:25.267580 22653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:06:25.268507 22653 raft_consensus.cc:686] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [term 1 LEADER]: Becoming Leader. State: Replica: b18c7fd46b7e4ed686ea29edc78c5242, State: Running, Role: LEADER
I20260620 23:06:25.269105 22653 consensus_queue.cc:237] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18c7fd46b7e4ed686ea29edc78c5242" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40965 } }
I20260620 23:06:25.269907 22648 sys_catalog.cc:564] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:06:25.271955 22655 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18c7fd46b7e4ed686ea29edc78c5242. Latest consensus state: current_term: 1 leader_uuid: "b18c7fd46b7e4ed686ea29edc78c5242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18c7fd46b7e4ed686ea29edc78c5242" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40965 } } }
I20260620 23:06:25.272300 22655 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:25.273329 22654 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b18c7fd46b7e4ed686ea29edc78c5242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18c7fd46b7e4ed686ea29edc78c5242" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40965 } } }
I20260620 23:06:25.273800 22654 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:25.274705 22659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:06:25.279896 22659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:06:25.288033 22659 catalog_manager.cc:1347] Generated new cluster ID: ca3cb90ce12a47be8fac781b4ddd8e47
I20260620 23:06:25.288147 22659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:06:25.300123 22659 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:06:25.302016 22659 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:06:25.311610 22659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242: Generated new TSK 0
I20260620 23:06:25.312655 22659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:06:25.373798 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40965
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:25.683463 22674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:25.683748 22674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:25.683884 22674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:25.691032 22674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:25.691154 22674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:25.691305 22674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:06:25.700820 22674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:25.702315 22674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:25.703634 22674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:25.715499 22679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:25.715662 22680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:25.715917 22682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:25.716566 22674 server_base.cc:1034] running on GCE node
I20260620 23:06:25.717114 22674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:25.718019 22674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:25.719200 22674 hybrid_clock.cc:648] HybridClock initialized: now 1781996785719153 us; error 40 us; skew 500 ppm
I20260620 23:06:25.719480 22674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:06:25 dist-test-slave-7g8g krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:25.722450 22674 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:06:25.723847 22674 webserver.cc:460] Webserver started at http://127.12.2.65:34239/ using document root <none> and password file <none>
I20260620 23:06:25.724500 22674 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:25.724620 22674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:25.724879 22674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:25.727682 22674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb"
format_stamp: "Formatted at 2026-06-20 23:06:25 on dist-test-slave-7g8g"
server_key: "a1e01c9c8b88c0d780058ef0dabe0082"
server_key_iv: "8423bb5817fb2e64164fb0da5270c05a"
server_key_version: "encryptionkey@0"
I20260620 23:06:25.728366 22674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb"
format_stamp: "Formatted at 2026-06-20 23:06:25 on dist-test-slave-7g8g"
server_key: "a1e01c9c8b88c0d780058ef0dabe0082"
server_key_iv: "8423bb5817fb2e64164fb0da5270c05a"
server_key_version: "encryptionkey@0"
I20260620 23:06:25.733144 22674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:06:25.736480 22689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:25.737735 22674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:06:25.737877 22674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb"
format_stamp: "Formatted at 2026-06-20 23:06:25 on dist-test-slave-7g8g"
server_key: "a1e01c9c8b88c0d780058ef0dabe0082"
server_key_iv: "8423bb5817fb2e64164fb0da5270c05a"
server_key_version: "encryptionkey@0"
I20260620 23:06:25.738039 22674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:25.755363 22674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:25.759018 22674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:25.759263 22674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:25.760140 22674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:25.761404 22674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:25.761482 22674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:25.761605 22674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:25.761657 22674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:25.790689 22674 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:41879
I20260620 23:06:25.790732 22801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:41879 every 8 connection(s)
I20260620 23:06:25.791925 22674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:06:25.796164 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 22674
I20260620 23:06:25.796397 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:06:25.797256 12297 external_mini_cluster.cc:1428] Setting key 8bca36b6a1a2eafdaa2fa4daf0942aa8
Jun 20 23:06:25 dist-test-slave-7g8g krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996785, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:06:25.813527 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:25.794263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:51245 (local address 127.12.2.126:40965)
0620 23:06:25.794676 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:25.794681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:25.795569 (+ 888us) server_negotiation.cc:407] Connection header received
0620 23:06:25.798125 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:25.798131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:25.798221 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:25.798326 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:25.800183 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:25.800910 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:25.803016 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:25.803468 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:25.803569 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:25.807476 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:25.807482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:25.807487 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:25.807525 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:25.810355 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:25.811108 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:25.811114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:25.811118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:25.811279 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:25.811830 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:25.811836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:25.811840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:25.812157 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:25.812276 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:25.813059 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 23:06:25.813230 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":205,"threads_started":1}
I20260620 23:06:25.814806 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:25.794650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:40965 (local address 127.12.2.65:51245)
0620 23:06:25.795390 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:25.795445 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:25.797819 (+ 2374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:25.798595 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:25.798604 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:25.799184 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:25.800005 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:25.800033 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:25.801063 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:25.801070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:25.802840 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:25.802855 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:25.803683 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:25.803690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:25.803969 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:25.804622 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:25.804652 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:25.807226 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:25.810503 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:25.810510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:25.810522 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:25.810926 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:25.811420 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:25.811425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:25.811429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:25.811683 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:25.812270 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:25.812278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:25.812672 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 23:06:25.812934 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 23:06:25.813413 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":306,"threads_started":1}
I20260620 23:06:25.816362 22802 heartbeater.cc:347] Connected to a master server at 127.12.2.126:40965
I20260620 23:06:25.816648 22802 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:25.817396 22802 heartbeater.cc:511] Master 127.12.2.126:40965 requested a full tablet report, sending...
I20260620 23:06:25.819403 22613 ts_manager.cc:194] Registered new tserver with Master: d6122d3680fd4ce9b8a5a443a17aa1cb (127.12.2.65:41879)
I20260620 23:06:25.820791 22613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:51245
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:06:25.846117 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40965
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:26.151000 22808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:26.151345 22808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:26.151469 22808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:26.158452 22808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:26.158593 22808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:26.158712 22808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:06:26.167949 22808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:26.169157 22808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:26.170400 22808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:26.182843 22813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:26.182767 22816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:26.182848 22814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:26.183528 22808 server_base.cc:1034] running on GCE node
I20260620 23:06:26.184064 22808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:26.184901 22808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:26.186100 22808 hybrid_clock.cc:648] HybridClock initialized: now 1781996786186054 us; error 40 us; skew 500 ppm
I20260620 23:06:26.186395 22808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:06:26 dist-test-slave-7g8g krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996786, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:26.189323 22808 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:06:26.192998 22808 webserver.cc:460] Webserver started at http://127.12.2.66:36683/ using document root <none> and password file <none>
I20260620 23:06:26.193756 22808 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:26.193850 22808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:26.194077 22808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:26.196789 22808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "e1b94a0c2f8b46148f11c31cd34fca4d"
format_stamp: "Formatted at 2026-06-20 23:06:26 on dist-test-slave-7g8g"
server_key: "232ce41bba6cfd9ca1a1b79ee7ebf783"
server_key_iv: "fd751f3be9dcf3c51644a1bec505d8f9"
server_key_version: "encryptionkey@0"
I20260620 23:06:26.197440 22808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "e1b94a0c2f8b46148f11c31cd34fca4d"
format_stamp: "Formatted at 2026-06-20 23:06:26 on dist-test-slave-7g8g"
server_key: "232ce41bba6cfd9ca1a1b79ee7ebf783"
server_key_iv: "fd751f3be9dcf3c51644a1bec505d8f9"
server_key_version: "encryptionkey@0"
I20260620 23:06:26.202219 22808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 23:06:26.205435 22823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:26.206689 22808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:06:26.206823 22808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "e1b94a0c2f8b46148f11c31cd34fca4d"
format_stamp: "Formatted at 2026-06-20 23:06:26 on dist-test-slave-7g8g"
server_key: "232ce41bba6cfd9ca1a1b79ee7ebf783"
server_key_iv: "fd751f3be9dcf3c51644a1bec505d8f9"
server_key_version: "encryptionkey@0"
I20260620 23:06:26.207024 22808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:26.229310 22808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:26.233239 22808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:26.233497 22808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:26.234411 22808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:26.235802 22808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:26.235881 22808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:26.236021 22808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:26.236074 22808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:26.263770 22808 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:37489
I20260620 23:06:26.263813 22935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:37489 every 8 connection(s)
I20260620 23:06:26.264951 22808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:06:26.274848 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 22808
I20260620 23:06:26.275069 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:06:26.275606 12297 external_mini_cluster.cc:1428] Setting key 0906ce319046d7b68b8b9db4cdc1dda9
Jun 20 23:06:26 dist-test-slave-7g8g krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996786, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:06:26.284592 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.267354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:36341 (local address 127.12.2.126:40965)
0620 23:06:26.267521 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:26.267525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:26.268714 (+ 1189us) server_negotiation.cc:407] Connection header received
0620 23:06:26.270645 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:26.270649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:26.270712 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:26.270795 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:26.272355 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.273067 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.274628 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.275224 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.275343 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.278850 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:26.278857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:26.278861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:26.278899 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:26.281717 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:26.282404 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:26.282409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:26.282412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:26.282556 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:26.282964 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:26.282968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:26.282971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:26.283258 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:26.283390 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:26.284083 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 23:06:26.284288 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:06:26.285566 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.267817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:40965 (local address 127.12.2.66:36341)
0620 23:06:26.268550 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:26.268590 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:26.270387 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:26.271116 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:26.271129 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:26.271646 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:26.272218 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.272235 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.273213 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.273218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.274475 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.274486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.275449 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.275459 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.275686 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.276403 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:26.276431 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:26.278635 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:26.281847 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:26.281854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:26.281866 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:26.282302 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:26.282658 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:26.282662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:26.282664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:26.282847 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:26.283388 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:26.283395 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:26.283707 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 23:06:26.283993 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 23:06:26.284385 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":288,"threads_started":1}
I20260620 23:06:26.287038 22936 heartbeater.cc:347] Connected to a master server at 127.12.2.126:40965
I20260620 23:06:26.287355 22936 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:26.288151 22936 heartbeater.cc:511] Master 127.12.2.126:40965 requested a full tablet report, sending...
I20260620 23:06:26.289726 22613 ts_manager.cc:194] Registered new tserver with Master: e1b94a0c2f8b46148f11c31cd34fca4d (127.12.2.66:37489)
I20260620 23:06:26.290761 22613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:36341
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:06:26.323643 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40965
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:26.644074 22941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:26.644497 22941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:26.644676 22941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:26.652619 22941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:26.652735 22941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:26.652899 22941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:06:26.662860 22941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:06:26.664186 22941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:26.665462 22941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:26.679908 22947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:26.680104 22946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:26.680274 22949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:26.681025 22941 server_base.cc:1034] running on GCE node
I20260620 23:06:26.681623 22941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:26.682600 22941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:26.683802 22941 hybrid_clock.cc:648] HybridClock initialized: now 1781996786683736 us; error 54 us; skew 500 ppm
I20260620 23:06:26.684110 22941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:06:26 dist-test-slave-7g8g krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996786, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:26.687180 22941 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:06:26.688630 22941 webserver.cc:460] Webserver started at http://127.12.2.67:37159/ using document root <none> and password file <none>
I20260620 23:06:26.689277 22941 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:26.689385 22941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:26.689670 22941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:26.692482 22941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "1dfeeca3d7cc4d25a049acb883fbd332"
format_stamp: "Formatted at 2026-06-20 23:06:26 on dist-test-slave-7g8g"
server_key: "35b28b73dcf84b85dd464a6754080c9f"
server_key_iv: "01912209ca4419be119ab0dedfe78973"
server_key_version: "encryptionkey@0"
I20260620 23:06:26.693181 22941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "1dfeeca3d7cc4d25a049acb883fbd332"
format_stamp: "Formatted at 2026-06-20 23:06:26 on dist-test-slave-7g8g"
server_key: "35b28b73dcf84b85dd464a6754080c9f"
server_key_iv: "01912209ca4419be119ab0dedfe78973"
server_key_version: "encryptionkey@0"
I20260620 23:06:26.698243 22941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:06:26.701728 22956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:26.703307 22941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:06:26.703493 22941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "1dfeeca3d7cc4d25a049acb883fbd332"
format_stamp: "Formatted at 2026-06-20 23:06:26 on dist-test-slave-7g8g"
server_key: "35b28b73dcf84b85dd464a6754080c9f"
server_key_iv: "01912209ca4419be119ab0dedfe78973"
server_key_version: "encryptionkey@0"
I20260620 23:06:26.703675 22941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:26.723637 22941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:26.727794 22941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:26.728061 22941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:26.729028 22941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:06:26.730337 22941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:06:26.730412 22941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:26.730535 22941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:06:26.730585 22941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:26.767920 22941 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:34865
I20260620 23:06:26.767952 23068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:34865 every 8 connection(s)
I20260620 23:06:26.769166 22941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:06:26.774158 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 22941
I20260620 23:06:26.774340 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:06:26.774876 12297 external_mini_cluster.cc:1428] Setting key 1f98a159f6d261aff76c604d7e2226b5
Jun 20 23:06:26 dist-test-slave-7g8g krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996786, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:06:26.795760 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.771750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:42511 (local address 127.12.2.126:40965)
0620 23:06:26.771949 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:26.771957 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:26.773193 (+ 1236us) server_negotiation.cc:407] Connection header received
0620 23:06:26.775529 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:26.775534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:26.775614 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:26.775703 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:26.777803 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.778631 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.780154 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.780785 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.780913 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.788768 (+ 7855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:26.788773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:26.788776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:26.788814 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:26.792125 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:26.793070 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:26.793076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:26.793079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:26.793223 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:26.793714 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:26.793720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:26.793724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:26.794095 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:26.794233 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:26.795233 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0620 23:06:26.795449 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:06:26.797055 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.772237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:40965 (local address 127.12.2.67:42511)
0620 23:06:26.773033 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:26.773079 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:26.775204 (+ 2125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:26.776055 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:26.776066 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:26.776824 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:26.777594 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.777619 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.778782 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.778786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.780020 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.780031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.780969 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.780974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.781168 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.786048 (+ 4880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:26.786076 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:26.788542 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:26.792279 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:26.792290 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:26.792310 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:26.792927 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:26.793326 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:26.793332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:26.793336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:26.793580 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:26.794322 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:26.794330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:26.794769 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 23:06:26.795129 (+ 360us) client_negotiation.cc:240] Negotiation successful
0620 23:06:26.795685 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":329,"threads_started":1}
I20260620 23:06:26.798550 23069 heartbeater.cc:347] Connected to a master server at 127.12.2.126:40965
I20260620 23:06:26.798831 23069 heartbeater.cc:464] Registering TS with master...
I20260620 23:06:26.799588 23069 heartbeater.cc:511] Master 127.12.2.126:40965 requested a full tablet report, sending...
I20260620 23:06:26.801047 22613 ts_manager.cc:194] Registered new tserver with Master: 1dfeeca3d7cc4d25a049acb883fbd332 (127.12.2.67:34865)
I20260620 23:06:26.802243 22613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:42511
I20260620 23:06:26.802965 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:06:26.821990 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.807940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.12.2.126:40965)
0620 23:06:26.808111 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:26.808116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:26.808724 (+ 608us) server_negotiation.cc:407] Connection header received
0620 23:06:26.808881 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:26.808885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:26.808944 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:26.809027 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:26.810848 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.811674 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.813472 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.813867 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.813954 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.816779 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:26.816785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:26.816789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:26.816829 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:26.819526 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:26.820287 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:26.820291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:26.820293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:26.820415 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:26.820764 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:26.820766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:26.820768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:26.821057 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:26.821229 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:26.821560 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 23:06:26.821715 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:06:26.823585 22802 heartbeater.cc:503] Master 127.12.2.126:40965 was elected leader, sending a full tablet report...
I20260620 23:06:26.826864 22612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:06:26.835310 22612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:06:26.867765 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.847463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:41879 (local address 127.0.0.1:36826)
0620 23:06:26.849124 (+ 1661us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:26.849166 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:26.849337 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:26.860338 (+ 11001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:26.860346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:26.860375 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:06:26.860737 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.860748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.862683 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.862688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.864958 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.864972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.866705 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.866712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.866901 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.866984 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 23:06:26.867108 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 23:06:26.867294 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1395,"mutex_wait_us":5,"thread_start_us":2113,"threads_started":1}
I20260620 23:06:26.868122 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.850031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:37489 (local address 127.0.0.1:37858)
0620 23:06:26.859971 (+ 9940us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:26.859987 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:26.860097 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:26.861093 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:26.861099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:26.861128 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:06:26.861545 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.861557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.863214 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.863219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.865391 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.865406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.867220 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.867227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.867406 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.867442 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:06:26.867545 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 23:06:26.867687 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9821,"mutex_wait_us":60,"thread_start_us":9706,"threads_started":1}
I20260620 23:06:26.868233 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.847945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.12.2.65:41879)
0620 23:06:26.859686 (+ 11741us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:26.859695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:26.859726 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:06:26.859825 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:26.859832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:26.860110 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:26.860912 (+ 802us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:06:26.861143 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.862505 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.865127 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.866515 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.866664 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.866914 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:26.867804 (+ 890us) server_negotiation.cc:299] Negotiation successful
0620 23:06:26.867966 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11580,"thread_start_us":11433,"threads_started":1}
I20260620 23:06:26.868569 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.847882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:34865 (local address 127.0.0.1:50836)
0620 23:06:26.849848 (+ 1966us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:26.849864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:26.850342 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:26.862066 (+ 11724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:26.862072 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:26.862096 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:06:26.862470 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.862484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.864373 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.864378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.866082 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.866092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.867583 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.867590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.867779 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.867818 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:06:26.868100 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 23:06:26.868234 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1848,"thread_start_us":843,"threads_started":1}
I20260620 23:06:26.869071 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.850510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.12.2.66:37489)
0620 23:06:26.860525 (+ 10015us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:26.860534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:26.860560 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:06:26.860644 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:26.860651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:26.860871 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:26.861055 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:06:26.861733 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.863034 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.865556 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.867005 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.867146 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.867342 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:26.868334 (+ 992us) server_negotiation.cc:299] Negotiation successful
0620 23:06:26.868505 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9867,"thread_start_us":9758,"threads_started":1}
I20260620 23:06:26.869279 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.848475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.12.2.67:34865)
0620 23:06:26.861559 (+ 13084us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:26.861568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:26.861604 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:06:26.861699 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:26.861706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:26.861882 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:26.862063 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:06:26.862637 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.864206 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.866215 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.867376 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.867499 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.867694 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:26.868849 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0620 23:06:26.868989 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12905,"thread_start_us":13801,"threads_started":1}
I20260620 23:06:26.872300 22737 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8225e671e049dc86abf745f17e58c2 (DEFAULT_TABLE table=test-table [id=42a10986235344fda942fcb2be7fdb2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:06:26.872709 22871 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8225e671e049dc86abf745f17e58c2 (DEFAULT_TABLE table=test-table [id=42a10986235344fda942fcb2be7fdb2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:06:26.872720 23004 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8225e671e049dc86abf745f17e58c2 (DEFAULT_TABLE table=test-table [id=42a10986235344fda942fcb2be7fdb2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:06:26.873742 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8225e671e049dc86abf745f17e58c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:26.874022 22871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8225e671e049dc86abf745f17e58c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:26.874038 23004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8225e671e049dc86abf745f17e58c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:26.886637 23087 tablet_bootstrap.cc:492] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb: Bootstrap starting.
I20260620 23:06:26.887734 23088 tablet_bootstrap.cc:492] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: Bootstrap starting.
I20260620 23:06:26.890765 23087 tablet_bootstrap.cc:654] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:26.891042 23088 tablet_bootstrap.cc:654] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:26.891652 23089 tablet_bootstrap.cc:492] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: Bootstrap starting.
I20260620 23:06:26.892261 23088 log.cc:824] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:26.892275 23087 log.cc:824] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:26.894872 23089 tablet_bootstrap.cc:654] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:26.895012 23087 tablet_bootstrap.cc:492] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb: No bootstrap required, opened a new log
I20260620 23:06:26.895268 23087 ts_tablet_manager.cc:1397] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260620 23:06:26.895362 23088 tablet_bootstrap.cc:492] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: No bootstrap required, opened a new log
I20260620 23:06:26.895604 23088 ts_tablet_manager.cc:1397] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 23:06:26.896065 23089 log.cc:824] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:26.899225 23089 tablet_bootstrap.cc:492] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: No bootstrap required, opened a new log
I20260620 23:06:26.899475 23089 ts_tablet_manager.cc:1397] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 23:06:26.902808 23087 raft_consensus.cc:348] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 41879 } } peers { permanent_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34865 } } peers { permanent_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 37489 } }
I20260620 23:06:26.903200 23087 raft_consensus.cc:374] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:06:26.903302 23087 raft_consensus.cc:729] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6122d3680fd4ce9b8a5a443a17aa1cb, State: Initialized, Role: FOLLOWER
I20260620 23:06:26.903858 23087 consensus_queue.cc:260] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 41879 } } peers { permanent_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34865 } } peers { permanent_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 37489 } }
I20260620 23:06:26.905637 23087 ts_tablet_manager.cc:1428] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260620 23:06:26.905401 23088 raft_consensus.cc:348] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 41879 } } peers { permanent_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34865 } } peers { permanent_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 37489 } }
I20260620 23:06:26.905771 23088 raft_consensus.cc:374] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:06:26.905898 23088 raft_consensus.cc:729] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b94a0c2f8b46148f11c31cd34fca4d, State: Initialized, Role: FOLLOWER
I20260620 23:06:26.906572 23088 consensus_queue.cc:260] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 41879 } } peers { permanent_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34865 } } peers { permanent_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 37489 } }
I20260620 23:06:26.908159 23088 ts_tablet_manager.cc:1428] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260620 23:06:26.908667 22936 heartbeater.cc:503] Master 127.12.2.126:40965 was elected leader, sending a full tablet report...
I20260620 23:06:26.908999 23089 raft_consensus.cc:348] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 41879 } } peers { permanent_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34865 } } peers { permanent_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 37489 } }
I20260620 23:06:26.909288 23089 raft_consensus.cc:374] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:06:26.909345 23089 raft_consensus.cc:729] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dfeeca3d7cc4d25a049acb883fbd332, State: Initialized, Role: FOLLOWER
I20260620 23:06:26.909809 23089 consensus_queue.cc:260] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 41879 } } peers { permanent_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34865 } } peers { permanent_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 37489 } }
I20260620 23:06:26.912137 23069 heartbeater.cc:503] Master 127.12.2.126:40965 was elected leader, sending a full tablet report...
I20260620 23:06:26.912627 23089 ts_tablet_manager.cc:1428] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260620 23:06:26.968199 23095 raft_consensus.cc:482] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:06:26.968470 23095 raft_consensus.cc:504] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 41879 } } peers { permanent_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34865 } } peers { permanent_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 37489 } }
I20260620 23:06:26.970350 23095 leader_election.cc:288] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6122d3680fd4ce9b8a5a443a17aa1cb (127.12.2.65:41879), e1b94a0c2f8b46148f11c31cd34fca4d (127.12.2.66:37489)
I20260620 23:06:26.976478 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.970647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:41879 (local address 127.12.2.67:49109)
0620 23:06:26.970863 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:26.970880 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:26.971010 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:26.971365 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:26.971371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:26.971395 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:06:26.971588 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.971599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.972760 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.972764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.974577 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.974593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.975793 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.975800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.976051 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.976078 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:06:26.976158 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:06:26.976253 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260620 23:06:26.976581 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.970755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:49109 (local address 127.12.2.65:41879)
0620 23:06:26.970996 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:26.971002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:26.971022 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:06:26.971159 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:26.971165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:26.971258 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:26.971337 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:06:26.971724 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.972612 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.974738 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.975633 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.975760 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.975860 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:26.976200 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 23:06:26.976328 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:06:26.977406 22757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8225e671e049dc86abf745f17e58c2" candidate_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" is_pre_election: true
I20260620 23:06:26.977573 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.971176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:42287 (local address 127.12.2.66:37489)
0620 23:06:26.971353 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:26.971359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:26.971879 (+ 520us) server_negotiation.cc:407] Connection header received
0620 23:06:26.971986 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:26.971991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:26.972097 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:26.972187 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:06:26.972619 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.973311 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.975473 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.976556 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.976677 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.976764 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:26.977183 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 23:06:26.977311 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:06:26.977648 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:26.971081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:37489 (local address 127.12.2.67:42287)
0620 23:06:26.971788 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:26.971800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:26.971892 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:26.972205 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:26.972208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:26.972226 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:06:26.972487 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.972494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.973487 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.973493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.975326 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:26.975338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:26.976879 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:26.976886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:26.977023 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:26.977047 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:06:26.977136 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:06:26.977255 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":1,"thread_start_us":444,"threads_started":1}
I20260620 23:06:26.977919 22757 raft_consensus.cc:2454] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dfeeca3d7cc4d25a049acb883fbd332 in term 0.
I20260620 23:06:26.978567 22959 leader_election.cc:302] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dfeeca3d7cc4d25a049acb883fbd332, d6122d3680fd4ce9b8a5a443a17aa1cb; no voters:
I20260620 23:06:26.978560 22891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8225e671e049dc86abf745f17e58c2" candidate_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" is_pre_election: true
I20260620 23:06:26.978916 23095 raft_consensus.cc:2781] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:06:26.979105 23095 raft_consensus.cc:482] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:06:26.979223 23095 raft_consensus.cc:3037] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:26.979171 22891 raft_consensus.cc:2454] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dfeeca3d7cc4d25a049acb883fbd332 in term 0.
I20260620 23:06:26.981531 23095 raft_consensus.cc:504] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 41879 } } peers { permanent_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34865 } } peers { permanent_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 37489 } }
I20260620 23:06:26.982118 23095 leader_election.cc:288] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [CANDIDATE]: Term 1 election: Requested vote from peers d6122d3680fd4ce9b8a5a443a17aa1cb (127.12.2.65:41879), e1b94a0c2f8b46148f11c31cd34fca4d (127.12.2.66:37489)
I20260620 23:06:26.982503 22757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8225e671e049dc86abf745f17e58c2" candidate_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb"
I20260620 23:06:26.982571 22891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8225e671e049dc86abf745f17e58c2" candidate_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d"
I20260620 23:06:26.982667 22757 raft_consensus.cc:3037] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:26.982764 22891 raft_consensus.cc:3037] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:26.984745 22757 raft_consensus.cc:2454] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dfeeca3d7cc4d25a049acb883fbd332 in term 1.
I20260620 23:06:26.985034 22891 raft_consensus.cc:2454] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dfeeca3d7cc4d25a049acb883fbd332 in term 1.
I20260620 23:06:26.985074 22959 leader_election.cc:302] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dfeeca3d7cc4d25a049acb883fbd332, d6122d3680fd4ce9b8a5a443a17aa1cb; no voters:
I20260620 23:06:26.985349 23095 raft_consensus.cc:2781] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:06:26.985826 23095 raft_consensus.cc:686] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 1 LEADER]: Becoming Leader. State: Replica: 1dfeeca3d7cc4d25a049acb883fbd332, State: Running, Role: LEADER
I20260620 23:06:26.986526 23095 consensus_queue.cc:237] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 41879 } } peers { permanent_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34865 } } peers { permanent_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 37489 } }
I20260620 23:06:26.991183 22613 catalog_manager.cc:5495] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dfeeca3d7cc4d25a049acb883fbd332 (127.12.2.67). New cstate: current_term: 1 leader_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 41879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dfeeca3d7cc4d25a049acb883fbd332" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 37489 } health_report { overall_health: UNKNOWN } } }
I20260620 23:06:27.010639 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:27.005404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.12.2.67:34865)
0620 23:06:27.005612 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:27.005617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:27.005636 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:06:27.005799 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:27.005803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:27.005873 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:27.005995 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:06:27.006401 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:27.007293 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:27.008623 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:27.009026 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:27.009130 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:27.009625 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:06:27.010103 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:06:27.010228 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:27.010315 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:06:27.010386 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 23:06:27.016669 22937 tablet.cc:2369] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:06:27.018687 22891 raft_consensus.cc:1261] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d [term 1 FOLLOWER]: Refusing update from remote peer 1dfeeca3d7cc4d25a049acb883fbd332: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:06:27.018736 22757 raft_consensus.cc:1261] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb [term 1 FOLLOWER]: Refusing update from remote peer 1dfeeca3d7cc4d25a049acb883fbd332: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:06:27.019521 23097 consensus_queue.cc:1035] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1b94a0c2f8b46148f11c31cd34fca4d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:06:27.019843 23095 consensus_queue.cc:1035] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6122d3680fd4ce9b8a5a443a17aa1cb" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 41879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:06:27.020798 23070 tablet.cc:2369] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:06:27.030097 23101 mvcc.cc:205] Tried to move back new op lower bound from 7299058839616786432 to 7299058839500795904. Current Snapshot: MvccSnapshot[applied={T|T < 7299058839616786432}]
I20260620 23:06:27.035852 23102 mvcc.cc:205] Tried to move back new op lower bound from 7299058839616786432 to 7299058839500795904. Current Snapshot: MvccSnapshot[applied={T|T < 7299058839616786432}]
I20260620 23:06:27.042251 22613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:06:27.042500 22613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:06:27.045698 22613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b18c7fd46b7e4ed686ea29edc78c5242: Sending DeleteTablet for 3 replicas of tablet 6e8225e671e049dc86abf745f17e58c2
I20260620 23:06:27.046682 23004 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e8225e671e049dc86abf745f17e58c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:06:27 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:50836
I20260620 23:06:27.047024 22737 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e8225e671e049dc86abf745f17e58c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:06:27 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:36826
I20260620 23:06:27.047350 22871 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e8225e671e049dc86abf745f17e58c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:06:27 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:37858
I20260620 23:06:27.047899 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 22674
I20260620 23:06:27.062103 23112 tablet_replica.cc:330] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: stopping tablet replica
I20260620 23:06:27.062695 23112 raft_consensus.cc:2229] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:06:27.063398 23112 raft_consensus.cc:2258] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:06:27.065186 23110 tablet_replica.cc:330] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: stopping tablet replica
I20260620 23:06:27.065733 23110 raft_consensus.cc:2229] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:06:27.066388 23110 raft_consensus.cc:2258] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:06:27.068646 23112 ts_tablet_manager.cc:1905] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:06:27.068655 23110 ts_tablet_manager.cc:1905] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:06:27.071382 23111 tablet_replica.cc:330] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb: stopping tablet replica
I20260620 23:06:27.071811 23111 raft_consensus.cc:2229] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:06:27.072352 23111 raft_consensus.cc:2258] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:06:27.074416 23111 ts_tablet_manager.cc:1905] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:06:27.076649 23110 ts_tablet_manager.cc:1918] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:06:27.076854 23110 log.cc:1196] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/6e8225e671e049dc86abf745f17e58c2
I20260620 23:06:27.077432 23110 ts_tablet_manager.cc:1939] T 6e8225e671e049dc86abf745f17e58c2 P 1dfeeca3d7cc4d25a049acb883fbd332: Deleting consensus metadata
Jun 20 23:06:27 dist-test-slave-7g8g krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:06:27.079401 22598 catalog_manager.cc:4841] TS 1dfeeca3d7cc4d25a049acb883fbd332 (127.12.2.67:34865): tablet 6e8225e671e049dc86abf745f17e58c2 (table test-table [id=42a10986235344fda942fcb2be7fdb2a]) successfully deleted
I20260620 23:06:27.081283 23111 ts_tablet_manager.cc:1918] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:06:27.081523 23111 log.cc:1196] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/6e8225e671e049dc86abf745f17e58c2
I20260620 23:06:27.082062 23111 ts_tablet_manager.cc:1939] T 6e8225e671e049dc86abf745f17e58c2 P d6122d3680fd4ce9b8a5a443a17aa1cb: Deleting consensus metadata
I20260620 23:06:27.082600 23112 ts_tablet_manager.cc:1918] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:06:27.082791 23112 log.cc:1196] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/6e8225e671e049dc86abf745f17e58c2
I20260620 23:06:27.083323 23112 ts_tablet_manager.cc:1939] T 6e8225e671e049dc86abf745f17e58c2 P e1b94a0c2f8b46148f11c31cd34fca4d: Deleting consensus metadata
I20260620 23:06:27.084149 22599 catalog_manager.cc:4841] TS d6122d3680fd4ce9b8a5a443a17aa1cb (127.12.2.65:41879): tablet 6e8225e671e049dc86abf745f17e58c2 (table test-table [id=42a10986235344fda942fcb2be7fdb2a]) successfully deleted
I20260620 23:06:27.084929 22599 catalog_manager.cc:4841] TS e1b94a0c2f8b46148f11c31cd34fca4d (127.12.2.66:37489): tablet 6e8225e671e049dc86abf745f17e58c2 (table test-table [id=42a10986235344fda942fcb2be7fdb2a]) successfully deleted
I20260620 23:06:27.086858 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:27.049315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.12.2.65:41879)
0620 23:06:27.049478 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:27.049484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:27.059251 (+ 9767us) server_negotiation.cc:407] Connection header received
0620 23:06:27.059703 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:27.059711 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:27.059797 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:27.059866 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:27.063624 (+ 3758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:27.064863 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:27.068867 (+ 4004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:27.069493 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:27.069602 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:27.079953 (+ 10351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:27.079971 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:27.079990 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:27.080043 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:27.084018 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:27.084994 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:27.085003 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:27.085006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:27.085155 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:27.085553 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:27.085557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:27.085559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:27.085902 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:27.086119 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:27.086446 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 23:06:27.086611 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:06:27.087632 22797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:27.174465 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 22674
I20260620 23:06:27.188545 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 22808
Jun 20 23:06:27 dist-test-slave-7g8g krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:06:27.204453 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:27.189268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.12.2.66:37489)
0620 23:06:27.189510 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:27.189516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:27.189536 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:06:27.189656 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:27.189661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:27.189748 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:27.189853 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:27.191118 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:27.192213 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:27.193404 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:27.194001 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:27.194125 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:27.197892 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:27.197911 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:27.197933 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:27.197976 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:27.201585 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:27.202296 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:27.202305 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:27.202309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:27.202451 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:27.202770 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:27.202775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:27.202779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:27.203279 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:27.203568 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:27.203846 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 23:06:27.204072 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 23:06:27.205196 22931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:27.292845 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 22808
I20260620 23:06:27.310608 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 22941
Jun 20 23:06:27 dist-test-slave-7g8g krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:06:27.325467 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:27.311407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.12.2.67:34865)
0620 23:06:27.311600 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:27.311605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:27.311624 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:06:27.311754 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:27.311759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:27.311835 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:27.311920 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:27.313157 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:27.313994 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:27.315274 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:27.315713 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:27.315823 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:27.319801 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:27.319813 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:27.319827 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:27.319860 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:27.322782 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:27.323497 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:27.323504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:27.323506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:27.323677 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:27.324018 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:27.324022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:27.324024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:27.324358 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:27.324675 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:27.324900 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 23:06:27.325103 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:06:27.326133 23064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:27.442871 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 22941
I20260620 23:06:27.457978 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 22581
I20260620 23:06:27.459030 22643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:06:27.562812 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 22581
2026-06-20T23:06:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3982 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:06:27 dist-test-slave-7g8g krb5kdc[23120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:06:27 dist-test-slave-7g8g krb5kdc[23120](info): set up 2 sockets
Jun 20 23:06:27 dist-test-slave-7g8g krb5kdc[23120](info): commencing operation
krb5kdc: starting...
W20260620 23:06:28.567449 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:06:28 dist-test-slave-7g8g krb5kdc[23120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:06:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/rangeradmin_127.12.2.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/rangeradmin_127.12.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/rangerlookup_127.12.2.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/rangerlookup_127.12.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/HTTP_127.12.2.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/HTTP_127.12.2.84@KRBTEST.COM.keytab.
I20260620 23:06:28.862180 12297 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyvnGin/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/postgres -l logfile start
2026-06-20 23:06:31.147 UTC [23150] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:06:31.147 UTC [23150] LOG: listening on IPv4 address "127.12.2.84", port 38479
2026-06-20 23:06:31.151 UTC [23150] LOG: listening on Unix socket "/tmp/.s.PGSQL.38479"
2026-06-20 23:06:31.172 UTC [23153] LOG: database system was shut down at 2026-06-20 23:06:29 UTC
2026-06-20 23:06:31.175 UTC [23150] LOG: database system is ready to accept connections
I20260620 23:06:32.067214 12297 mini_postgres.cc:96] Postgres bound to 38479
2026-06-20 23:06:32.083 UTC [23161] FATAL: database "slave" does not exist
127.12.2.84:38479 - accepting connections
I20260620 23:06:32.084677 12297 mini_ranger.cc:161] Starting Ranger...
I20260620 23:06:32.109412 12297 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:06:32.544790 12297 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:06:32.544914 12297 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-admin
2026-06-20 23:06:32,818 [I] DB FLAVOR :POSTGRES
2026-06-20 23:06:32,818 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:06:32,819 [I] Checking connection..
2026-06-20 23:06:32,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:06:33,051 [I] Checking connection passed.
2026-06-20 23:06:33,051 [I] --------- Verifying version history table ---------
2026-06-20 23:06:33,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:06:33,284 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:06:33,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:06:33,518 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:06:33,518 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:06:33,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:06:33.762 UTC [23225] WARNING: there is no transaction in progress
2026-06-20 23:06:33,767 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:06:33,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:06:34,012 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:06:34,012 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:06:34,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:06:34,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:06:34,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7g8g.c.gcp-upstream.internal','N') ;"
2026-06-20 23:06:34,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:06:34,951 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:06:34,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:06:35,195 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:06:35,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:06:35,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:06:35,657 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:06:35,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:06:36.576 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.591 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.605 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.618 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.632 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.648 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.701 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.707 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.720 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.730 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.739 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.753 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.762 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.768 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.775 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:36.781 UTC [23333] WARNING: there is no transaction in progress
2026-06-20 23:06:37,359 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:06:37,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:06:37,605 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:06:37,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:06:37,840 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:06:37,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:06:38,095 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:06:38,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:06:38,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:06:38,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7g8g.c.gcp-upstream.internal';"
2026-06-20 23:06:38,803 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:06:38,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:06:39,044 [I] DB_PATCHES have already been applied
I20260620 23:06:39.049618 12297 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyvnGin/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:06:39.049724 12297 mini_ranger.cc:193] Using host: 127.12.2.84
I20260620 23:06:39.063398 12297 mini_ranger.cc:227] Ranger admin URL: http://127.12.2.84:52283
Jun 20, 2026 11:06:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:06:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:06:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:06:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:06:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:06:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:06:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:06:40.078081 12297 mini_ranger.cc:160] Time spent starting Ranger: real 7.993s user 0.000s sys 0.037s
Jun 20, 2026 11:06:58 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:06:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 11:06:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:06:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:06:59 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:07:00.468667 12297 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/rangerkms_127.12.2.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/rangerkms_127.12.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.2.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.2.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.2.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/HTTP_127.12.2.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.2.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/HTTP_127.12.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:07:00 dist-test-slave-7g8g krb5kdc[23120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996820, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:07:00.631174 12297 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:07:00.650782 12297 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:07:01.397393 12297 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:07:01.397549 12297 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-kms
I20260620 23:07:01.397694 12297 mini_ranger_kms.cc:227] Using postgres at 127.12.2.84:38479
2026-06-20 23:07:01,796 [I] DB FLAVOR :POSTGRES
2026-06-20 23:07:01,797 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:07:01,797 [I] Checking connection
2026-06-20 23:07:01,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:07:02,022 [I] connection success
2026-06-20 23:07:02,023 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:07:02,023 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:07:02,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:07:02,276 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:07:02,276 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:07:02,276 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:07:02,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:38479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:07:02,533 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:07:03.018829 12297 mini_ranger_kms.cc:310] Created kms service
I20260620 23:07:03.190018 12297 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:07:03.232554 12297 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:07:03 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996820, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:03.611171 12297 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:07:03.611464 12297 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:07:03.611526 12297 mini_ranger_kms.cc:240] Using host: 127.12.2.84
I20260620 23:07:03.631966 12297 mini_ranger_kms.cc:276] Ranger KMS PID: 23513
I20260620 23:07:03.632126 12297 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.2.84:52551
Jun 20, 2026 11:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 23:07:05 dist-test-slave-7g8g krb5kdc[23120](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996825, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.2.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:07:05.692945 12297 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.062s user 0.129s sys 0.292s
I20260620 23:07:05.693210 12297 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:07:05.693434 12297 mini_ranger_kms.cc:401] 127.12.2.84:52551/kms/v1/keys
Jun 20 23:07:05 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996825, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.2.84@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
Jun 20 23:07:06 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996825, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.2.84@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
Jun 20 23:07:08 dist-test-slave-7g8g krb5kdc[23120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:07:08.606365 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:39443
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:39443
--ranger_config_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:52551/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:07:08.937150 23564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:07:08.937434 23564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:07:08.937547 23564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:07:08.944691 23564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:07:08.944803 23564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:07:08.944856 23564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:07:08.944936 23564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:07:08.944980 23564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:07:08.955353 23564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.2.84:52551/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:39443
--ranger_config_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:39443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:07:08.956609 23564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:07:08.957892 23564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:07:08.971884 23569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:08.971824 23570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:08.971902 23572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:07:08.972303 23564 server_base.cc:1034] running on GCE node
I20260620 23:07:08.973376 23564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:07:08.975184 23564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:07:08.976425 23564 hybrid_clock.cc:648] HybridClock initialized: now 1781996828976376 us; error 42 us; skew 500 ppm
I20260620 23:07:08.976840 23564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:07:08 dist-test-slave-7g8g krb5kdc[23120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996828, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:07:08.984124 23564 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:07:08.985778 23564 webserver.cc:460] Webserver started at http://127.12.2.126:41309/ using document root <none> and password file <none>
I20260620 23:07:08.986426 23564 fs_manager.cc:359] Metadata directory not provided
I20260620 23:07:08.986541 23564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:07:08.986836 23564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:07:08 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996828, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:09.035007 23564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "947a98cac5aa4440a6524f279d344227"
format_stamp: "Formatted at 2026-06-20 23:07:09 on dist-test-slave-7g8g"
server_key: "6c444a95b23118de1a27ce91b553a957"
server_key_iv: "46e04b0864f5e2aae0fe54766afdbcd2"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:09.035841 23564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "947a98cac5aa4440a6524f279d344227"
format_stamp: "Formatted at 2026-06-20 23:07:09 on dist-test-slave-7g8g"
server_key: "6c444a95b23118de1a27ce91b553a957"
server_key_iv: "46e04b0864f5e2aae0fe54766afdbcd2"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:09.041333 23564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:07:09.057785 23579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:09.059342 23564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:07:09.059506 23564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "947a98cac5aa4440a6524f279d344227"
format_stamp: "Formatted at 2026-06-20 23:07:09 on dist-test-slave-7g8g"
server_key: "6c444a95b23118de1a27ce91b553a957"
server_key_iv: "46e04b0864f5e2aae0fe54766afdbcd2"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:09.059712 23564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:07:09.075906 23564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:07:09.079800 23564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:07:09.080090 23564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:07:09.099529 23564 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:39443
I20260620 23:07:09.099546 23630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:39443 every 8 connection(s)
I20260620 23:07:09.101001 23564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:07:09.105341 23631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:07:09.111390 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 23564
I20260620 23:07:09.111599 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:07:09.112414 12297 external_mini_cluster.cc:1428] Setting key 466e60bf981b32f4300de4bb9f79837d
I20260620 23:07:09.115990 23631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227: Bootstrap starting.
I20260620 23:07:09.119400 23631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227: Neither blocks nor log segments found. Creating new log.
I20260620 23:07:09.120375 23631 log.cc:824] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227: Log is configured to *not* fsync() on all Append() calls
I20260620 23:07:09.123235 23631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227: No bootstrap required, opened a new log
Jun 20 23:07:09 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:07:09.130700 23631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947a98cac5aa4440a6524f279d344227" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39443 } }
I20260620 23:07:09.131093 23631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:07:09.131194 23631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947a98cac5aa4440a6524f279d344227, State: Initialized, Role: FOLLOWER
I20260620 23:07:09.131855 23631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947a98cac5aa4440a6524f279d344227" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39443 } }
I20260620 23:07:09.132124 23631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:07:09.132253 23631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:07:09.132428 23631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:07:09.133455 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:09.114478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.12.2.126:39443)
0620 23:07:09.115244 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:09.115256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:09.115293 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:07:09.116856 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:09.116878 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:09.117279 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:09.117607 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:09.119088 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:09.120410 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:09.121842 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:09.122505 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:09.122629 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:09.126305 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:09.126313 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:09.126327 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:09.126363 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:09.129213 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:09.129858 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:09.129864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:09.129874 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:09.130032 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:09.130389 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:09.130400 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:09.130402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:09.130908 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:09.131207 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:09.131630 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 23:07:09.132077 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":375,"threads_started":1}
I20260620 23:07:09.135171 23631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947a98cac5aa4440a6524f279d344227" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39443 } }
I20260620 23:07:09.136049 23631 leader_election.cc:302] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 947a98cac5aa4440a6524f279d344227; no voters:
I20260620 23:07:09.136677 23631 leader_election.cc:288] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:07:09.136809 23636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:07:09.137548 23636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [term 1 LEADER]: Becoming Leader. State: Replica: 947a98cac5aa4440a6524f279d344227, State: Running, Role: LEADER
I20260620 23:07:09.138059 23636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947a98cac5aa4440a6524f279d344227" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39443 } }
I20260620 23:07:09.138729 23631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:07:09.140635 23637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "947a98cac5aa4440a6524f279d344227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947a98cac5aa4440a6524f279d344227" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39443 } } }
I20260620 23:07:09.141001 23637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [sys.catalog]: This master's current role is: LEADER
I20260620 23:07:09.141624 23638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 947a98cac5aa4440a6524f279d344227. Latest consensus state: current_term: 1 leader_uuid: "947a98cac5aa4440a6524f279d344227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947a98cac5aa4440a6524f279d344227" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 39443 } } }
I20260620 23:07:09.141950 23638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227 [sys.catalog]: This master's current role is: LEADER
I20260620 23:07:09.146597 23641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:07:09.146741 23631 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:07:09.159791 23641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:07:09.174049 23641 catalog_manager.cc:1347] Generated new cluster ID: 14ce6a64667f4e22bf8f2763ac027e86
I20260620 23:07:09.174193 23641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:07:09.196249 23641 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:07:09.198288 23641 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:07:09.213544 23641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227: Generated new TSK 0
I20260620 23:07:09.214870 23641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:07:09.941334 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:52551/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39443
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:07:10 dist-test-slave-7g8g krb5kdc[23120](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996830, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:07:10.380043 23676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:07:10.380441 23676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:07:10.380602 23676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:07:10.392499 23676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:07:10.392638 23676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:07:10.392895 23676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:07:10.409129 23676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.2.84:52551/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:07:10.411427 23676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:07:10.413127 23676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:07:10.433771 23686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:10.434288 23687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:10.437143 23689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:07:10.437546 23676 server_base.cc:1034] running on GCE node
I20260620 23:07:10.438205 23676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:07:10.443004 23676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:07:10.444226 23676 hybrid_clock.cc:648] HybridClock initialized: now 1781996830444153 us; error 68 us; skew 500 ppm
I20260620 23:07:10.444538 23676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:07:10 dist-test-slave-7g8g krb5kdc[23120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996830, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:07:10.448159 23676 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:07:10.449748 23676 webserver.cc:460] Webserver started at http://127.12.2.65:36329/ using document root <none> and password file <none>
I20260620 23:07:10.450448 23676 fs_manager.cc:359] Metadata directory not provided
I20260620 23:07:10.450562 23676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:07:10.450840 23676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:07:10 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996830, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:10.472316 23676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "1e04e113b0d743a2ae73b2a7e43f0147"
format_stamp: "Formatted at 2026-06-20 23:07:10 on dist-test-slave-7g8g"
server_key: "2030a5d9fd59bb4b8b7d00318c1b11b3"
server_key_iv: "50782a99947563d1de56d9b460f29872"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:10.473724 23676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "1e04e113b0d743a2ae73b2a7e43f0147"
format_stamp: "Formatted at 2026-06-20 23:07:10 on dist-test-slave-7g8g"
server_key: "2030a5d9fd59bb4b8b7d00318c1b11b3"
server_key_iv: "50782a99947563d1de56d9b460f29872"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:10.485615 23676 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.003s sys 0.007s
I20260620 23:07:10.521931 23696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:10.523718 23676 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.007s sys 0.003s
I20260620 23:07:10.523905 23676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "1e04e113b0d743a2ae73b2a7e43f0147"
format_stamp: "Formatted at 2026-06-20 23:07:10 on dist-test-slave-7g8g"
server_key: "2030a5d9fd59bb4b8b7d00318c1b11b3"
server_key_iv: "50782a99947563d1de56d9b460f29872"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:10.524123 23676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:07:10.540361 23676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:07:10.544735 23676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:07:10.545015 23676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:07:10.546487 23676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:07:10.551641 23676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:07:10.551723 23676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:10.551877 23676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:07:10.551929 23676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:10.596071 23676 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:37721
I20260620 23:07:10.597332 23808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:37721 every 8 connection(s)
I20260620 23:07:10.597775 23676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:07:10.606549 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 23676
I20260620 23:07:10.606766 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:07:10.607635 12297 external_mini_cluster.cc:1428] Setting key 0a1a8ff3d7739161a1572a1ba6313b99
Jun 20 23:07:10 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996830, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Jun 20 23:07:10 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996830, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:10.664702 23811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:10.601182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:58397 (local address 127.12.2.126:39443)
0620 23:07:10.619514 (+ 18332us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:10.619521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:10.620515 (+ 994us) server_negotiation.cc:407] Connection header received
0620 23:07:10.623128 (+ 2613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:10.623134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:10.623232 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:10.623338 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:10.625708 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:10.626827 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:10.631609 (+ 4782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:10.632271 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:10.632372 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:10.654353 (+ 21981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:10.654361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:10.654366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:10.654430 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:10.658353 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:10.660385 (+ 2032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:10.660391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:10.660395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:10.660546 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:10.661503 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:10.661508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:10.661511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:10.661846 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:10.661981 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:10.664074 (+ 2093us) server_negotiation.cc:299] Negotiation successful
0620 23:07:10.664323 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18188,"thread_start_us":18074,"threads_started":1}
I20260620 23:07:10.667095 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:10.601791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:39443 (local address 127.12.2.65:58397)
0620 23:07:10.620175 (+ 18384us) negotiation.cc:107] Waiting for socket to connect
0620 23:07:10.620244 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 23:07:10.622595 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:07:10.623804 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:07:10.623816 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:07:10.624471 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:07:10.625487 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:10.625518 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:10.629333 (+ 3815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:10.629338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:10.631420 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:10.631437 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:10.632713 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:10.632719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:10.633031 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:10.633787 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:07:10.633818 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:07:10.651388 (+ 17570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:07:10.659480 (+ 8092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:10.659491 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:10.659514 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:10.660211 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:10.661043 (+ 832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:10.661048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:10.661052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:10.661366 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:10.662068 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:07:10.662077 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:07:10.662556 (+ 479us) client_negotiation.cc:769] Sending connection context
0620 23:07:10.664944 (+ 2388us) client_negotiation.cc:240] Negotiation successful
0620 23:07:10.665569 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18047,"thread_start_us":17813,"threads_started":1}
I20260620 23:07:10.668932 23809 heartbeater.cc:347] Connected to a master server at 127.12.2.126:39443
I20260620 23:07:10.669252 23809 heartbeater.cc:464] Registering TS with master...
I20260620 23:07:10.670074 23809 heartbeater.cc:511] Master 127.12.2.126:39443 requested a full tablet report, sending...
I20260620 23:07:10.672575 23596 ts_manager.cc:194] Registered new tserver with Master: 1e04e113b0d743a2ae73b2a7e43f0147 (127.12.2.65:37721)
I20260620 23:07:10.677953 23596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:58397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:07:10.693897 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:52551/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39443
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:07:10 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996830, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
W20260620 23:07:11.160549 23815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:07:11.161033 23815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:07:11.161231 23815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:07:11.173141 23815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:07:11.173283 23815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:07:11.173524 23815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:07:11.189556 23815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.2.84:52551/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:07:11.191511 23815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:07:11.193174 23815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:07:11.240341 23822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:11.241602 23823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:11.245646 23825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:07:11.249718 23815 server_base.cc:1034] running on GCE node
I20260620 23:07:11.250753 23815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:07:11.252226 23815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:07:11.255649 23815 hybrid_clock.cc:648] HybridClock initialized: now 1781996831255530 us; error 104 us; skew 500 ppm
I20260620 23:07:11.256098 23815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:07:11 dist-test-slave-7g8g krb5kdc[23120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:07:11.260321 23815 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:07:11.262363 23815 webserver.cc:460] Webserver started at http://127.12.2.66:36701/ using document root <none> and password file <none>
I20260620 23:07:11.263306 23815 fs_manager.cc:359] Metadata directory not provided
I20260620 23:07:11.263432 23815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:07:11.263764 23815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:07:11 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:11.288154 23815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "4989f519f7464f3d83eeaad5e21c703f"
format_stamp: "Formatted at 2026-06-20 23:07:11 on dist-test-slave-7g8g"
server_key: "5dc9dd12b77d440ab70a19de2c646bb8"
server_key_iv: "5d135091625b93c1320655f10bb85971"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:11.289620 23815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "4989f519f7464f3d83eeaad5e21c703f"
format_stamp: "Formatted at 2026-06-20 23:07:11 on dist-test-slave-7g8g"
server_key: "5dc9dd12b77d440ab70a19de2c646bb8"
server_key_iv: "5d135091625b93c1320655f10bb85971"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:11.313225 23815 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.010s sys 0.011s
I20260620 23:07:11.331920 23839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:11.333369 23815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:07:11.333536 23815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "4989f519f7464f3d83eeaad5e21c703f"
format_stamp: "Formatted at 2026-06-20 23:07:11 on dist-test-slave-7g8g"
server_key: "5dc9dd12b77d440ab70a19de2c646bb8"
server_key_iv: "5d135091625b93c1320655f10bb85971"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:11.333740 23815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:07:11.354113 23815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:07:11.358280 23815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:07:11.358549 23815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:07:11.359601 23815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:07:11.360981 23815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:07:11.361049 23815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:11.361143 23815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:07:11.361197 23815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:11.395037 23815 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:33019
I20260620 23:07:11.395053 23951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:33019 every 8 connection(s)
I20260620 23:07:11.396340 23815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:07:11.405557 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 23815
I20260620 23:07:11.405745 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:07:11.406267 12297 external_mini_cluster.cc:1428] Setting key 77e3f7389d576e209d2033f4064e4192
Jun 20 23:07:11 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:07:11.421152 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:11.399200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:59361 (local address 127.12.2.126:39443)
0620 23:07:11.399692 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:11.399699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:11.400793 (+ 1094us) server_negotiation.cc:407] Connection header received
0620 23:07:11.402614 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:11.402620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:11.402700 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:11.402838 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:11.404181 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:11.405310 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:11.407506 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:11.408142 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:11.408260 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:11.412938 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:11.412945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:11.412950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:11.412989 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:11.417533 (+ 4544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:11.418511 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:11.418518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:11.418521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:11.418671 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:11.419215 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:11.419220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:11.419224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:11.419554 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:11.419692 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:11.420543 (+ 851us) server_negotiation.cc:299] Negotiation successful
0620 23:07:11.420763 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260620 23:07:11.422358 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:11.399835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:39443 (local address 127.12.2.66:59361)
0620 23:07:11.400609 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 23:07:11.400665 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:07:11.402342 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:07:11.403059 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:07:11.403067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:07:11.403461 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:07:11.404012 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:11.404029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:11.405488 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:11.405494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:11.407349 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:11.407371 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:11.408435 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:11.408454 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:11.408762 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:11.409502 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:07:11.409529 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:07:11.412659 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:07:11.417729 (+ 5070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:11.417739 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:11.417759 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:11.418360 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:11.418785 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:11.418790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:11.418793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:11.419094 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:11.419692 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:07:11.419701 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:07:11.420123 (+ 422us) client_negotiation.cc:769] Sending connection context
0620 23:07:11.420415 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 23:07:11.420926 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":325,"threads_started":1}
I20260620 23:07:11.423840 23952 heartbeater.cc:347] Connected to a master server at 127.12.2.126:39443
I20260620 23:07:11.424186 23952 heartbeater.cc:464] Registering TS with master...
I20260620 23:07:11.424960 23952 heartbeater.cc:511] Master 127.12.2.126:39443 requested a full tablet report, sending...
I20260620 23:07:11.426664 23596 ts_manager.cc:194] Registered new tserver with Master: 4989f519f7464f3d83eeaad5e21c703f (127.12.2.66:33019)
I20260620 23:07:11.427742 23596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:59361
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:07:11.460525 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:52551/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39443
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:07:11.681619 23809 heartbeater.cc:503] Master 127.12.2.126:39443 was elected leader, sending a full tablet report...
W20260620 23:07:11.789853 23958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:07:11.790174 23958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:07:11.790318 23958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:07:11.797533 23958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:07:11.797663 23958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:07:11.797819 23958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:07:11.807713 23958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.2.84:52551/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:39443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:07:11.809016 23958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:07:11.810245 23958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:07:11.822062 23963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:11.822013 23966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:11.822099 23964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:07:11.823104 23958 server_base.cc:1034] running on GCE node
I20260620 23:07:11.823698 23958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:07:11.824599 23958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:07:11.825793 23958 hybrid_clock.cc:648] HybridClock initialized: now 1781996831825746 us; error 45 us; skew 500 ppm
I20260620 23:07:11.826100 23958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:07:11 dist-test-slave-7g8g krb5kdc[23120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:07:11.829582 23958 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:07:11.830864 23958 webserver.cc:460] Webserver started at http://127.12.2.67:41973/ using document root <none> and password file <none>
I20260620 23:07:11.831506 23958 fs_manager.cc:359] Metadata directory not provided
I20260620 23:07:11.831604 23958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:07:11.831880 23958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:07:11 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:11.847354 23958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "14017b6465c241aabd0b9c01bf2c4c10"
format_stamp: "Formatted at 2026-06-20 23:07:11 on dist-test-slave-7g8g"
server_key: "4004a0d206ee7047ebe0b7d4a6c4921f"
server_key_iv: "ce490ff5c07bb7a5765d3d26f55c9e66"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:11.848342 23958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "14017b6465c241aabd0b9c01bf2c4c10"
format_stamp: "Formatted at 2026-06-20 23:07:11 on dist-test-slave-7g8g"
server_key: "4004a0d206ee7047ebe0b7d4a6c4921f"
server_key_iv: "ce490ff5c07bb7a5765d3d26f55c9e66"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:11.854610 23958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 23:07:11.871170 23973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:11.872828 23958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:07:11.873018 23958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "14017b6465c241aabd0b9c01bf2c4c10"
format_stamp: "Formatted at 2026-06-20 23:07:11 on dist-test-slave-7g8g"
server_key: "4004a0d206ee7047ebe0b7d4a6c4921f"
server_key_iv: "ce490ff5c07bb7a5765d3d26f55c9e66"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:11.873263 23958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:07:11.889971 23958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:07:11.894284 23958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:07:11.894584 23958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:07:11.895573 23958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:07:11.896950 23958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:07:11.897033 23958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:11.897161 23958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:07:11.897219 23958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:11.935294 23958 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:42953
I20260620 23:07:11.935345 24085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:42953 every 8 connection(s)
I20260620 23:07:11.936599 23958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:07:11.945133 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 23958
I20260620 23:07:11.945302 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:07:11.945688 12297 external_mini_cluster.cc:1428] Setting key 6a2e8af82cc45a6dc1ca9dfe8ceeb835
Jun 20 23:07:11 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:07:11.958639 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:11.940054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:41405 (local address 127.12.2.126:39443)
0620 23:07:11.940588 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:11.940594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:11.941530 (+ 936us) server_negotiation.cc:407] Connection header received
0620 23:07:11.943869 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:11.943875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:11.943955 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:11.944046 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:11.945869 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:11.946819 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:11.948836 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:11.949290 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:11.949402 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:11.952760 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:11.952765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:11.952769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:11.952800 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:11.955260 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:11.956369 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:11.956373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:11.956375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:11.956491 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:11.956874 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:11.956878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:11.956880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:11.957146 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:11.957282 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:11.958219 (+ 937us) server_negotiation.cc:299] Negotiation successful
0620 23:07:11.958390 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":215,"threads_started":1}
I20260620 23:07:11.960019 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:11.940545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:39443 (local address 127.12.2.67:41405)
0620 23:07:11.941334 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 23:07:11.941388 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:07:11.943587 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:07:11.944298 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:07:11.944310 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:07:11.944879 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:07:11.945654 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:11.945679 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:11.947032 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:11.947038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:11.948648 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:11.948663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:11.949502 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:11.949510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:11.949729 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:11.950390 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:07:11.950411 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:07:11.952553 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:07:11.955382 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:11.955391 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:11.955409 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:11.956242 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:11.956592 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:11.956597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:11.956601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:11.956763 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:11.957317 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:07:11.957325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:07:11.957768 (+ 443us) client_negotiation.cc:769] Sending connection context
0620 23:07:11.958098 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 23:07:11.958617 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":308,"threads_started":1}
I20260620 23:07:11.961615 24086 heartbeater.cc:347] Connected to a master server at 127.12.2.126:39443
I20260620 23:07:11.961920 24086 heartbeater.cc:464] Registering TS with master...
I20260620 23:07:11.962628 24086 heartbeater.cc:511] Master 127.12.2.126:39443 requested a full tablet report, sending...
I20260620 23:07:11.964068 23595 ts_manager.cc:194] Registered new tserver with Master: 14017b6465c241aabd0b9c01bf2c4c10 (127.12.2.67:42953)
I20260620 23:07:11.964890 23595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:41405
I20260620 23:07:11.973896 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:07:11.993824 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:11.978626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.12.2.126:39443)
0620 23:07:11.978815 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:11.978839 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:11.979361 (+ 522us) server_negotiation.cc:407] Connection header received
0620 23:07:11.979528 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:11.979532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:11.979614 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:11.979700 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:11.981491 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:11.982217 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:11.984219 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:11.984666 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:11.984773 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:11.987752 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:11.987758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:11.987761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:11.987798 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:11.991003 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:11.991904 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:11.991908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:11.991911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:11.992050 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:11.992471 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:11.992474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:11.992477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:11.992748 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:11.992940 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:11.993351 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 23:07:11.993545 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:07:11.998214 23595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:07:12.005599 23595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:07:12.047981 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.019623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:33019 (local address 127.0.0.1:39346)
0620 23:07:12.034426 (+ 14803us) negotiation.cc:107] Waiting for socket to connect
0620 23:07:12.034503 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 23:07:12.034709 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:07:12.038486 (+ 3777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:07:12.038495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:07:12.038530 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:07:12.039032 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:12.039046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.042174 (+ 3128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.042179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:12.045057 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:12.045071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.047086 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.047091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:12.047275 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.047322 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 23:07:12.047427 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:07:12.047627 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14634,"thread_start_us":14512,"threads_started":1}
I20260620 23:07:12.049198 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.019846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.12.2.66:33019)
0620 23:07:12.037477 (+ 17631us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:12.037488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:12.037522 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:07:12.037647 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:12.037656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:12.037956 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:12.038175 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:07:12.040585 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.042024 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.045236 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.046657 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.046770 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.048702 (+ 1932us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:12.048773 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 23:07:12.048933 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17424,"thread_start_us":19339,"threads_started":1}
I20260620 23:07:12.050868 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.019835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:37721 (local address 127.0.0.1:38772)
0620 23:07:12.035258 (+ 15423us) negotiation.cc:107] Waiting for socket to connect
0620 23:07:12.035274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:07:12.035388 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:07:12.040775 (+ 5387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:07:12.040780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:07:12.040810 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:07:12.041190 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:12.041201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.044693 (+ 3492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.044700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:12.048152 (+ 3452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:12.048168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.050164 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.050171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:12.050346 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.050388 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:07:12.050448 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:07:12.050575 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15302,"thread_start_us":15280,"threads_started":1}
I20260620 23:07:12.052042 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.019282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.12.2.65:37721)
0620 23:07:12.040173 (+ 20891us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:12.040182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:12.040208 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:07:12.040295 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:12.040302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:12.040547 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:12.040752 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:07:12.041395 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.044472 (+ 3077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.048390 (+ 3918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.049940 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.051221 (+ 1281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.051519 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:12.051587 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 23:07:12.051741 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20724,"thread_start_us":21645,"threads_started":1}
I20260620 23:07:12.053319 23887 tablet_service.cc:1467] Processing CreateTablet for tablet dad5a17f1d194cb09913acddd70354a3 (DEFAULT_TABLE table=test-table [id=efb90c1de6bc46b1b7986c20d11de41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:07:12.054509 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad5a17f1d194cb09913acddd70354a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:07:12.055579 23744 tablet_service.cc:1467] Processing CreateTablet for tablet dad5a17f1d194cb09913acddd70354a3 (DEFAULT_TABLE table=test-table [id=efb90c1de6bc46b1b7986c20d11de41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:07:12.056932 23744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad5a17f1d194cb09913acddd70354a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:07:12.063534 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.019803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.12.2.67:42953)
0620 23:07:12.055388 (+ 35585us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:12.055396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:12.055434 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:07:12.055525 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:12.055533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:12.055766 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:12.055974 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:07:12.056535 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.058062 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.060442 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.061857 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.061972 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.062171 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:12.063026 (+ 855us) server_negotiation.cc:299] Negotiation successful
0620 23:07:12.063191 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35410,"threads_started":1}
I20260620 23:07:12.063895 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.019158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:42953 (local address 127.0.0.1:55664)
0620 23:07:12.034426 (+ 15268us) negotiation.cc:107] Waiting for socket to connect
0620 23:07:12.034875 (+ 449us) client_negotiation.cc:174] Beginning negotiation
0620 23:07:12.035073 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:07:12.055976 (+ 20903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:07:12.055981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:07:12.056010 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:07:12.056310 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:12.056321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.058230 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.058235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:12.060241 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:12.060260 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.062307 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.062314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:12.062480 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.062512 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:07:12.062596 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:07:12.062723 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14895,"thread_start_us":15269,"threads_started":1}
I20260620 23:07:12.067737 24021 tablet_service.cc:1467] Processing CreateTablet for tablet dad5a17f1d194cb09913acddd70354a3 (DEFAULT_TABLE table=test-table [id=efb90c1de6bc46b1b7986c20d11de41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:07:12.069120 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad5a17f1d194cb09913acddd70354a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:07:12.076697 24105 tablet_bootstrap.cc:492] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: Bootstrap starting.
I20260620 23:07:12.080281 24105 tablet_bootstrap.cc:654] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: Neither blocks nor log segments found. Creating new log.
I20260620 23:07:12.081485 24105 log.cc:824] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:07:12.083452 24106 tablet_bootstrap.cc:492] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147: Bootstrap starting.
I20260620 23:07:12.087803 24108 tablet_bootstrap.cc:492] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: Bootstrap starting.
I20260620 23:07:12.089367 24106 tablet_bootstrap.cc:654] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147: Neither blocks nor log segments found. Creating new log.
I20260620 23:07:12.089498 24105 tablet_bootstrap.cc:492] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: No bootstrap required, opened a new log
I20260620 23:07:12.089762 24105 ts_tablet_manager.cc:1397] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260620 23:07:12.090764 24106 log.cc:824] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147: Log is configured to *not* fsync() on all Append() calls
I20260620 23:07:12.090894 24108 tablet_bootstrap.cc:654] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: Neither blocks nor log segments found. Creating new log.
I20260620 23:07:12.093751 24108 log.cc:824] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: Log is configured to *not* fsync() on all Append() calls
I20260620 23:07:12.100476 24106 tablet_bootstrap.cc:492] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147: No bootstrap required, opened a new log
I20260620 23:07:12.100785 24106 ts_tablet_manager.cc:1397] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
I20260620 23:07:12.100665 24105 raft_consensus.cc:348] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37721 } } peers { permanent_uuid: "14017b6465c241aabd0b9c01bf2c4c10" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 42953 } } peers { permanent_uuid: "4989f519f7464f3d83eeaad5e21c703f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 33019 } }
I20260620 23:07:12.101075 24105 raft_consensus.cc:374] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:07:12.101167 24105 raft_consensus.cc:729] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4989f519f7464f3d83eeaad5e21c703f, State: Initialized, Role: FOLLOWER
I20260620 23:07:12.101765 24105 consensus_queue.cc:260] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37721 } } peers { permanent_uuid: "14017b6465c241aabd0b9c01bf2c4c10" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 42953 } } peers { permanent_uuid: "4989f519f7464f3d83eeaad5e21c703f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 33019 } }
I20260620 23:07:12.109664 24105 ts_tablet_manager.cc:1428] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260620 23:07:12.110040 23952 heartbeater.cc:503] Master 127.12.2.126:39443 was elected leader, sending a full tablet report...
I20260620 23:07:12.111997 24106 raft_consensus.cc:348] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37721 } } peers { permanent_uuid: "14017b6465c241aabd0b9c01bf2c4c10" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 42953 } } peers { permanent_uuid: "4989f519f7464f3d83eeaad5e21c703f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 33019 } }
I20260620 23:07:12.112485 24106 raft_consensus.cc:374] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:07:12.112605 24106 raft_consensus.cc:729] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04e113b0d743a2ae73b2a7e43f0147, State: Initialized, Role: FOLLOWER
I20260620 23:07:12.113289 24106 consensus_queue.cc:260] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37721 } } peers { permanent_uuid: "14017b6465c241aabd0b9c01bf2c4c10" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 42953 } } peers { permanent_uuid: "4989f519f7464f3d83eeaad5e21c703f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 33019 } }
I20260620 23:07:12.121325 24106 ts_tablet_manager.cc:1428] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260620 23:07:12.121541 24108 tablet_bootstrap.cc:492] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: No bootstrap required, opened a new log
I20260620 23:07:12.121799 24108 ts_tablet_manager.cc:1397] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.000s
I20260620 23:07:12.129565 24108 raft_consensus.cc:348] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37721 } } peers { permanent_uuid: "14017b6465c241aabd0b9c01bf2c4c10" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 42953 } } peers { permanent_uuid: "4989f519f7464f3d83eeaad5e21c703f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 33019 } }
I20260620 23:07:12.129907 24108 raft_consensus.cc:374] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:07:12.129994 24108 raft_consensus.cc:729] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14017b6465c241aabd0b9c01bf2c4c10, State: Initialized, Role: FOLLOWER
I20260620 23:07:12.130537 24108 consensus_queue.cc:260] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37721 } } peers { permanent_uuid: "14017b6465c241aabd0b9c01bf2c4c10" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 42953 } } peers { permanent_uuid: "4989f519f7464f3d83eeaad5e21c703f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 33019 } }
I20260620 23:07:12.131613 24086 heartbeater.cc:503] Master 127.12.2.126:39443 was elected leader, sending a full tablet report...
I20260620 23:07:12.132068 24108 ts_tablet_manager.cc:1428] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
W20260620 23:07:12.148228 23953 tablet.cc:2369] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:07:12.189486 24087 tablet.cc:2369] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:07:12.297857 24112 raft_consensus.cc:482] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:07:12.298142 24112 raft_consensus.cc:504] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37721 } } peers { permanent_uuid: "14017b6465c241aabd0b9c01bf2c4c10" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 42953 } } peers { permanent_uuid: "4989f519f7464f3d83eeaad5e21c703f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 33019 } }
I20260620 23:07:12.299769 24112 leader_election.cc:288] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14017b6465c241aabd0b9c01bf2c4c10 (127.12.2.67:42953), 4989f519f7464f3d83eeaad5e21c703f (127.12.2.66:33019)
I20260620 23:07:12.306298 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.300202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:44459 (local address 127.12.2.66:33019)
0620 23:07:12.300391 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:12.300396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:12.301237 (+ 841us) server_negotiation.cc:407] Connection header received
0620 23:07:12.301465 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:12.301471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:12.301566 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:12.301651 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:07:12.302229 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.302919 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.304668 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.305394 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.305520 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.305609 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:12.305951 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 23:07:12.306068 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:07:12.306429 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.300195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:38655 (local address 127.12.2.67:42953)
0620 23:07:12.300475 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:12.300482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:12.301499 (+ 1017us) server_negotiation.cc:407] Connection header received
0620 23:07:12.301668 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:12.301673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:12.301731 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:12.301806 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:07:12.302427 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.303138 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.304777 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.305514 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.305618 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.305723 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:12.306074 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 23:07:12.306189 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:07:12.306456 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.300069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:33019 (local address 127.12.2.65:44459)
0620 23:07:12.301108 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0620 23:07:12.301128 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:07:12.301292 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:07:12.301704 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:07:12.301708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:07:12.301741 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:07:12.302080 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:12.302092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.303069 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.303074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:12.304508 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:12.304520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.305599 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.305606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:12.305771 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.305826 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 23:07:12.305908 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 23:07:12.306038 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":51,"thread_start_us":729,"threads_started":1}
I20260620 23:07:12.306545 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.300656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:42953 (local address 127.12.2.65:38655)
0620 23:07:12.301390 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 23:07:12.301404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:07:12.301544 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:07:12.301870 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:07:12.301874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:07:12.301903 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:07:12.302274 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:12.302283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.303274 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.303277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:12.304625 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:12.304637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.305724 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.305731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:12.305931 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.305954 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:07:12.306029 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:07:12.306144 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":67,"thread_start_us":476,"threads_started":1}
I20260620 23:07:12.307387 23907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad5a17f1d194cb09913acddd70354a3" candidate_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4989f519f7464f3d83eeaad5e21c703f" is_pre_election: true
I20260620 23:07:12.307410 24041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad5a17f1d194cb09913acddd70354a3" candidate_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14017b6465c241aabd0b9c01bf2c4c10" is_pre_election: true
I20260620 23:07:12.307816 24041 raft_consensus.cc:2454] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04e113b0d743a2ae73b2a7e43f0147 in term 0.
I20260620 23:07:12.307818 23907 raft_consensus.cc:2454] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e04e113b0d743a2ae73b2a7e43f0147 in term 0.
I20260620 23:07:12.308400 23699 leader_election.cc:302] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04e113b0d743a2ae73b2a7e43f0147, 4989f519f7464f3d83eeaad5e21c703f; no voters:
I20260620 23:07:12.308760 24112 raft_consensus.cc:2781] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:07:12.308887 24112 raft_consensus.cc:482] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:07:12.308984 24112 raft_consensus.cc:3037] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:07:12.311532 24112 raft_consensus.cc:504] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37721 } } peers { permanent_uuid: "14017b6465c241aabd0b9c01bf2c4c10" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 42953 } } peers { permanent_uuid: "4989f519f7464f3d83eeaad5e21c703f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 33019 } }
I20260620 23:07:12.312160 24112 leader_election.cc:288] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [CANDIDATE]: Term 1 election: Requested vote from peers 14017b6465c241aabd0b9c01bf2c4c10 (127.12.2.67:42953), 4989f519f7464f3d83eeaad5e21c703f (127.12.2.66:33019)
I20260620 23:07:12.312537 24041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad5a17f1d194cb09913acddd70354a3" candidate_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14017b6465c241aabd0b9c01bf2c4c10"
I20260620 23:07:12.312644 23907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad5a17f1d194cb09913acddd70354a3" candidate_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4989f519f7464f3d83eeaad5e21c703f"
I20260620 23:07:12.312743 24041 raft_consensus.cc:3037] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:07:12.312820 23907 raft_consensus.cc:3037] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:07:12.315788 24041 raft_consensus.cc:2454] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04e113b0d743a2ae73b2a7e43f0147 in term 1.
I20260620 23:07:12.315788 23907 raft_consensus.cc:2454] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e04e113b0d743a2ae73b2a7e43f0147 in term 1.
I20260620 23:07:12.316188 23699 leader_election.cc:302] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04e113b0d743a2ae73b2a7e43f0147, 4989f519f7464f3d83eeaad5e21c703f; no voters:
I20260620 23:07:12.316469 24112 raft_consensus.cc:2781] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:07:12.316892 24112 raft_consensus.cc:686] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 1 LEADER]: Becoming Leader. State: Replica: 1e04e113b0d743a2ae73b2a7e43f0147, State: Running, Role: LEADER
I20260620 23:07:12.317337 24112 consensus_queue.cc:237] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37721 } } peers { permanent_uuid: "14017b6465c241aabd0b9c01bf2c4c10" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 42953 } } peers { permanent_uuid: "4989f519f7464f3d83eeaad5e21c703f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 33019 } }
I20260620 23:07:12.321571 23595 catalog_manager.cc:5495] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e04e113b0d743a2ae73b2a7e43f0147 (127.12.2.65). New cstate: current_term: 1 leader_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04e113b0d743a2ae73b2a7e43f0147" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14017b6465c241aabd0b9c01bf2c4c10" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 42953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4989f519f7464f3d83eeaad5e21c703f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 33019 } health_report { overall_health: UNKNOWN } } }
I20260620 23:07:12.341162 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.335388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.12.2.65:37721)
0620 23:07:12.335577 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:12.335582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:12.335643 (+ 61us) server_negotiation.cc:407] Connection header received
0620 23:07:12.335889 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:12.335893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:12.335969 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:12.336092 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:07:12.336474 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.337203 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.338771 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.339648 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.339736 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.340192 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:07:12.340667 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:07:12.340770 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:12.340861 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 23:07:12.340924 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:07:12.349073 23907 raft_consensus.cc:1261] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f [term 1 FOLLOWER]: Refusing update from remote peer 1e04e113b0d743a2ae73b2a7e43f0147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:07:12.349074 24041 raft_consensus.cc:1261] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10 [term 1 FOLLOWER]: Refusing update from remote peer 1e04e113b0d743a2ae73b2a7e43f0147: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:07:12.349960 24116 consensus_queue.cc:1035] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14017b6465c241aabd0b9c01bf2c4c10" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 42953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:07:12.350343 24112 consensus_queue.cc:1035] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4989f519f7464f3d83eeaad5e21c703f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 33019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:07:12.363452 24120 mvcc.cc:205] Tried to move back new op lower bound from 7299059025292013568 to 7299059025175379968. Current Snapshot: MvccSnapshot[applied={T|T < 7299059025292013568}]
I20260620 23:07:12.377861 23595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:07:12.378163 23595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:07:12.383251 23595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 947a98cac5aa4440a6524f279d344227: Sending DeleteTablet for 3 replicas of tablet dad5a17f1d194cb09913acddd70354a3
I20260620 23:07:12.384490 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet dad5a17f1d194cb09913acddd70354a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:07:12 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:55664
I20260620 23:07:12.385279 24129 tablet_replica.cc:330] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: stopping tablet replica
I20260620 23:07:12.385762 24129 raft_consensus.cc:2229] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:07:12.385740 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet dad5a17f1d194cb09913acddd70354a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:07:12 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:39346
I20260620 23:07:12.385766 23744 tablet_service.cc:1514] Processing DeleteTablet for tablet dad5a17f1d194cb09913acddd70354a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:07:12 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:38772
I20260620 23:07:12.386370 24129 raft_consensus.cc:2258] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:07:12.386914 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet dad5a17f1d194cb09913acddd70354a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:07:12 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:39346
I20260620 23:07:12.387243 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet dad5a17f1d194cb09913acddd70354a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:07:12 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:55664
I20260620 23:07:12.388854 24129 ts_tablet_manager.cc:1905] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:07:12.389906 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 23676
I20260620 23:07:12.396332 24129 ts_tablet_manager.cc:1918] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:07:12.396606 24129 log.cc:1196] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/dad5a17f1d194cb09913acddd70354a3
I20260620 23:07:12.397140 24129 ts_tablet_manager.cc:1939] T dad5a17f1d194cb09913acddd70354a3 P 14017b6465c241aabd0b9c01bf2c4c10: Deleting consensus metadata
I20260620 23:07:12.398468 24131 tablet_replica.cc:330] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147: stopping tablet replica
I20260620 23:07:12.398867 24131 raft_consensus.cc:2229] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:07:12.399127 23583 catalog_manager.cc:4841] TS 14017b6465c241aabd0b9c01bf2c4c10 (127.12.2.67:42953): tablet dad5a17f1d194cb09913acddd70354a3 (table test-table [id=efb90c1de6bc46b1b7986c20d11de41b]) successfully deleted
I20260620 23:07:12.399587 24131 raft_consensus.cc:2258] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 23:07:12.400408 23583 catalog_manager.cc:4808] TS 14017b6465c241aabd0b9c01bf2c4c10 (127.12.2.67:42953): delete failed for tablet dad5a17f1d194cb09913acddd70354a3 because the tablet was not found. No further retry: Not found: Tablet not found: dad5a17f1d194cb09913acddd70354a3
I20260620 23:07:12.401279 24131 ts_tablet_manager.cc:1905] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:07:12.407313 24131 ts_tablet_manager.cc:1918] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:07:12.407460 24131 log.cc:1196] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/dad5a17f1d194cb09913acddd70354a3
I20260620 23:07:12.407984 24131 ts_tablet_manager.cc:1939] T dad5a17f1d194cb09913acddd70354a3 P 1e04e113b0d743a2ae73b2a7e43f0147: Deleting consensus metadata
I20260620 23:07:12.409641 23580 catalog_manager.cc:4841] TS 1e04e113b0d743a2ae73b2a7e43f0147 (127.12.2.65:37721): tablet dad5a17f1d194cb09913acddd70354a3 (table test-table [id=efb90c1de6bc46b1b7986c20d11de41b]) successfully deleted
I20260620 23:07:12.411425 24130 tablet_replica.cc:330] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: stopping tablet replica
I20260620 23:07:12.411861 24130 raft_consensus.cc:2229] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:07:12.412406 24130 raft_consensus.cc:2258] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:07:12.414499 24130 ts_tablet_manager.cc:1905] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:07:12 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:07:12.421423 24130 ts_tablet_manager.cc:1918] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:07:12.421567 24130 log.cc:1196] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/dad5a17f1d194cb09913acddd70354a3
I20260620 23:07:12.422029 24130 ts_tablet_manager.cc:1939] T dad5a17f1d194cb09913acddd70354a3 P 4989f519f7464f3d83eeaad5e21c703f: Deleting consensus metadata
I20260620 23:07:12.423710 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.391687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.12.2.65:37721)
0620 23:07:12.391873 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:12.391879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:12.403449 (+ 11570us) server_negotiation.cc:407] Connection header received
0620 23:07:12.403644 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:12.403649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:12.403725 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:12.403816 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:12.405924 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.407121 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.411289 (+ 4168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.411962 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.412055 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.417199 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:12.417213 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:12.417229 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:12.417272 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:12.420845 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:12.421495 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:12.421501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:12.421505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:12.421674 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:12.422040 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:12.422044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:12.422046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:12.422423 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:12.422679 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:12.423010 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 23:07:12.423236 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:07:12.423753 23582 catalog_manager.cc:4841] TS 4989f519f7464f3d83eeaad5e21c703f (127.12.2.66:33019): tablet dad5a17f1d194cb09913acddd70354a3 (table test-table [id=efb90c1de6bc46b1b7986c20d11de41b]) successfully deleted
W20260620 23:07:12.424274 23582 catalog_manager.cc:4808] TS 4989f519f7464f3d83eeaad5e21c703f (127.12.2.66:33019): delete failed for tablet dad5a17f1d194cb09913acddd70354a3 because the tablet was not found. No further retry: Not found: Tablet not found: dad5a17f1d194cb09913acddd70354a3
I20260620 23:07:12.424410 23804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:07:12.519662 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 23676
I20260620 23:07:12.534189 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 23815
Jun 20 23:07:12 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:07:12.548640 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.534923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.12.2.66:33019)
0620 23:07:12.535102 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:12.535107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:12.535125 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:07:12.535270 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:12.535274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:12.535350 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:12.535449 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:12.536627 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.537404 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.538720 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.539199 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.539296 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.543314 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:12.543329 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:12.543344 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:12.543393 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:12.546158 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:12.546722 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:12.546728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:12.546731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:12.546845 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:12.547340 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:12.547343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:12.547345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:12.547706 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:12.547932 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:12.548197 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 23:07:12.548351 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:07:12.549294 23947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:07:12.636049 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 23815
I20260620 23:07:12.648877 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 23958
Jun 20 23:07:12 dist-test-slave-7g8g krb5kdc[23120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:07:12.663295 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:12.649597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.12.2.67:42953)
0620 23:07:12.649745 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:12.649750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:12.649769 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:07:12.649960 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:12.649964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:12.650027 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:12.650116 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:12.651431 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.652195 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.653531 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:12.653955 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:12.654076 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:12.658186 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:12.658202 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:12.658217 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:12.658251 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:12.660817 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:12.661431 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:12.661439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:12.661441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:12.661562 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:12.661945 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:12.661950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:12.661952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:12.662304 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:12.662578 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:12.662769 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 23:07:12.663002 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:07:12.663898 24081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:07:12.749430 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 23958
I20260620 23:07:12.761698 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 23564
I20260620 23:07:12.762852 23626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:07:12.872470 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 23564
I20260620 23:07:12.888550 12297 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:07:13.223320 12297 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:07:13.223542 12297 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:07:13.568197 12297 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:07:13.568 UTC [23150] LOG: received smart shutdown request
2026-06-20 23:07:13.571 UTC [23150] LOG: background worker "logical replication launcher" (PID 23159) exited with exit code 1
2026-06-20 23:07:13.571 UTC [23154] LOG: shutting down
2026-06-20 23:07:13.716 UTC [23150] LOG: database system is shut down
2026-06-20T23:07:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46254 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:07:13 dist-test-slave-7g8g krb5kdc[24145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:07:13 dist-test-slave-7g8g krb5kdc[24145](info): set up 2 sockets
Jun 20 23:07:13 dist-test-slave-7g8g krb5kdc[24145](info): commencing operation
krb5kdc: starting...
W20260620 23:07:14.822885 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:07:14 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:07:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:07:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/rangeradmin_127.12.2.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/rangeradmin_127.12.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/rangerlookup_127.12.2.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/rangerlookup_127.12.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/HTTP_127.12.2.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/HTTP_127.12.2.84@KRBTEST.COM.keytab.
I20260620 23:07:15.128995 12297 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyvnGin/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/postgres -l logfile start
2026-06-20 23:07:17.476 UTC [24175] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:07:17.476 UTC [24175] LOG: listening on IPv4 address "127.12.2.84", port 50839
2026-06-20 23:07:17.486 UTC [24175] LOG: listening on Unix socket "/tmp/.s.PGSQL.50839"
2026-06-20 23:07:17.507 UTC [24178] LOG: database system was shut down at 2026-06-20 23:07:16 UTC
2026-06-20 23:07:17.511 UTC [24175] LOG: database system is ready to accept connections
I20260620 23:07:18.394661 12297 mini_postgres.cc:96] Postgres bound to 50839
2026-06-20 23:07:18.410 UTC [24186] FATAL: database "slave" does not exist
127.12.2.84:50839 - accepting connections
I20260620 23:07:18.411435 12297 mini_ranger.cc:161] Starting Ranger...
I20260620 23:07:18.435664 12297 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:07:18.891767 12297 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:07:18.891850 12297 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-admin
2026-06-20 23:07:19,153 [I] DB FLAVOR :POSTGRES
2026-06-20 23:07:19,153 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:07:19,153 [I] Checking connection..
2026-06-20 23:07:19,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:07:19,393 [I] Checking connection passed.
2026-06-20 23:07:19,393 [I] --------- Verifying version history table ---------
2026-06-20 23:07:19,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:07:19,635 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:07:19,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:07:19,875 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:07:19,876 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:07:19,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:07:20.119 UTC [24250] WARNING: there is no transaction in progress
2026-06-20 23:07:20,125 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:07:20,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:07:20,369 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:07:20,370 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:07:20,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:07:20,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:07:20,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7g8g.c.gcp-upstream.internal','N') ;"
2026-06-20 23:07:21,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:07:21,326 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:07:21,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:07:21,577 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:07:21,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:07:21,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:07:22,054 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:07:22,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:07:22.950 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:22.966 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:22.981 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:22.995 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.011 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.028 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.079 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.085 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.098 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.108 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.118 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.131 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.140 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.146 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.151 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23.158 UTC [24358] WARNING: there is no transaction in progress
2026-06-20 23:07:23,735 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:07:23,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:07:23,969 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:07:23,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:07:24,211 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:07:24,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:07:24,466 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:07:24,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:07:24,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:07:24,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7g8g.c.gcp-upstream.internal';"
2026-06-20 23:07:25,159 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:07:25,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:07:25,403 [I] DB_PATCHES have already been applied
I20260620 23:07:25.408631 12297 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyvnGin/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:07:25.408710 12297 mini_ranger.cc:193] Using host: 127.12.2.84
I20260620 23:07:25.422061 12297 mini_ranger.cc:227] Ranger admin URL: http://127.12.2.84:57581
Jun 20, 2026 11:07:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:07:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:07:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:07:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:07:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:07:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:07:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:07:26.437083 12297 mini_ranger.cc:160] Time spent starting Ranger: real 8.026s user 0.002s sys 0.036s
Jun 20, 2026 11:07:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:07:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 11:07:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:07:45 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:07:46 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:07:46.715045 12297 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.2.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.2.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/rangerkms_127.12.2.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.2.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/rangerkms_127.12.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.2.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.2.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.2.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/HTTP_127.12.2.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.2.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/HTTP_127.12.2.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:07:46 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:07:46.893174 12297 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:07:46.914956 12297 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:07:47.694994 12297 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:07:47.695096 12297 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-kms
I20260620 23:07:47.695137 12297 mini_ranger_kms.cc:227] Using postgres at 127.12.2.84:50839
2026-06-20 23:07:48,007 [I] DB FLAVOR :POSTGRES
2026-06-20 23:07:48,007 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:07:48,007 [I] Checking connection
2026-06-20 23:07:48,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:07:48,245 [I] connection success
2026-06-20 23:07:48,245 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:07:48,246 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:07:48,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:07:48,477 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:07:48,477 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:07:48,477 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:07:48,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.2.84:50839/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:07:48,731 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:07:49.179064 12297 mini_ranger_kms.cc:310] Created kms service
I20260620 23:07:49.357900 12297 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:07:49.402117 12297 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:07:49 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:49.768786 12297 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:07:49.769106 12297 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyvnGin/build/asan/bin/postgresql.jar:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyvnGin/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:07:49.769162 12297 mini_ranger_kms.cc:240] Using host: 127.12.2.84
I20260620 23:07:49.785537 12297 mini_ranger_kms.cc:276] Ranger KMS PID: 24538
I20260620 23:07:49.785670 12297 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.2.84:50887
Jun 20, 2026 11:07:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:07:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:07:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:07:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:07:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:07:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:07:51.866752 12297 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.974s user 0.066s sys 0.267s
I20260620 23:07:51.867061 12297 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:07:51.867259 12297 mini_ranger_kms.cc:401] 127.12.2.84:50887/kms/v1/keys
Jun 20 23:07:51 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996871, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.2.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:07:52 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996871, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.2.84@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
Jun 20 23:07:52 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996871, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.2.84@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
Jun 20 23:07:55 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:07:55.076270 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:37163
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:37163
--ranger_config_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:50887/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:07:55.384861 24589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:07:55.385133 24589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:07:55.385218 24589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:07:55.392165 24589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:07:55.392266 24589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:07:55.392313 24589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:07:55.392365 24589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:07:55.392413 24589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:07:55.402228 24589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.2.84:50887/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:37163
--ranger_config_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:37163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:07:55.403435 24589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:07:55.404664 24589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:07:55.415367 24594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:55.415341 24595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:55.415437 24597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:07:55.415989 24589 server_base.cc:1034] running on GCE node
I20260620 23:07:55.416661 24589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:07:55.418021 24589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:07:55.419205 24589 hybrid_clock.cc:648] HybridClock initialized: now 1781996875419154 us; error 47 us; skew 500 ppm
I20260620 23:07:55.419513 24589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:07:55 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996875, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:07:55.426704 24589 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:07:55.428181 24589 webserver.cc:460] Webserver started at http://127.12.2.126:39125/ using document root <none> and password file <none>
I20260620 23:07:55.428819 24589 fs_manager.cc:359] Metadata directory not provided
I20260620 23:07:55.428922 24589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:07:55.429198 24589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:07:55 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996875, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:55.477550 24589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "1fdf57ebc400408b977fbce4aec70cf3"
format_stamp: "Formatted at 2026-06-20 23:07:55 on dist-test-slave-7g8g"
server_key: "fcbbcb30c53b43e14ce1e32aa05e1a6b"
server_key_iv: "331ae177ee25a10f4ad24bca03d7a068"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:55.478364 24589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "1fdf57ebc400408b977fbce4aec70cf3"
format_stamp: "Formatted at 2026-06-20 23:07:55 on dist-test-slave-7g8g"
server_key: "fcbbcb30c53b43e14ce1e32aa05e1a6b"
server_key_iv: "331ae177ee25a10f4ad24bca03d7a068"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:55.483484 24589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 23:07:55.499730 24604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:55.501251 24589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:07:55.501418 24589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "1fdf57ebc400408b977fbce4aec70cf3"
format_stamp: "Formatted at 2026-06-20 23:07:55 on dist-test-slave-7g8g"
server_key: "fcbbcb30c53b43e14ce1e32aa05e1a6b"
server_key_iv: "331ae177ee25a10f4ad24bca03d7a068"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:55.501636 24589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:07:55.546675 24589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:07:55.550207 24589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:07:55.550480 24589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:07:55.566645 24589 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:37163
I20260620 23:07:55.566664 24655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:37163 every 8 connection(s)
I20260620 23:07:55.567946 24589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:07:55.571455 24656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:07:55.577922 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 24589
I20260620 23:07:55.578109 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:07:55.578742 12297 external_mini_cluster.cc:1428] Setting key d691e11aef1169cb66cbc9008a743041
I20260620 23:07:55.581035 24656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3: Bootstrap starting.
I20260620 23:07:55.584859 24656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3: Neither blocks nor log segments found. Creating new log.
I20260620 23:07:55.585981 24656 log.cc:824] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:07:55.588807 24656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3: No bootstrap required, opened a new log
Jun 20 23:07:55 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:07:55.597607 24656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } }
I20260620 23:07:55.598042 24656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:07:55.598145 24656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf57ebc400408b977fbce4aec70cf3, State: Initialized, Role: FOLLOWER
I20260620 23:07:55.598805 24656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } }
I20260620 23:07:55.599103 24656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:07:55.599241 24656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:07:55.599416 24656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:07:55.602340 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:55.580733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.12.2.126:37163)
0620 23:07:55.581566 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:55.581583 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:55.581915 (+ 332us) server_negotiation.cc:407] Connection header received
0620 23:07:55.584256 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:55.584282 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:55.584752 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:55.585178 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:55.586326 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:55.588108 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:55.589476 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:55.590454 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:55.590632 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:55.594526 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:55.594535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:55.594558 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:55.594605 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:55.598091 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:55.598769 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:55.598777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:55.598786 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:55.598992 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:55.599385 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:55.599391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:55.599394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:55.600080 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:55.600370 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:55.600695 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:07:55.601089 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":358,"threads_started":1}
I20260620 23:07:55.602473 24656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } }
I20260620 23:07:55.603024 24656 leader_election.cc:302] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fdf57ebc400408b977fbce4aec70cf3; no voters:
I20260620 23:07:55.603596 24656 leader_election.cc:288] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:07:55.603740 24661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:07:55.605207 24661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdf57ebc400408b977fbce4aec70cf3, State: Running, Role: LEADER
I20260620 23:07:55.605768 24661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } }
I20260620 23:07:55.605976 24656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:07:55.608523 24663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fdf57ebc400408b977fbce4aec70cf3. Latest consensus state: current_term: 1 leader_uuid: "1fdf57ebc400408b977fbce4aec70cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } } }
I20260620 23:07:55.608628 24662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fdf57ebc400408b977fbce4aec70cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } } }
I20260620 23:07:55.608927 24663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:07:55.608942 24662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:07:55.611105 24670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:07:55.616070 24670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:07:55.616796 24656 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:07:55.631747 24670 catalog_manager.cc:1347] Generated new cluster ID: d9e494a135ac498db8c12498c82c9b26
I20260620 23:07:55.631910 24670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:07:55.658025 24670 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:07:55.659468 24670 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:07:55.674270 24670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3: Generated new TSK 0
I20260620 23:07:55.675395 24670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:07:56.346344 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:50887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37163
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:07:56 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996876, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:07:56.737843 24701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:07:56.738119 24701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:07:56.738300 24701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:07:56.749457 24701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:07:56.749614 24701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:07:56.749811 24701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:07:56.764860 24701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.2.84:50887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:07:56.766543 24701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:07:56.768296 24701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:07:56.784821 24711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:56.784785 24712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:56.785670 24714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:07:56.785934 24701 server_base.cc:1034] running on GCE node
I20260620 23:07:56.786500 24701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:07:56.787523 24701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:07:56.788805 24701 hybrid_clock.cc:648] HybridClock initialized: now 1781996876788731 us; error 71 us; skew 500 ppm
I20260620 23:07:56.789096 24701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:07:56 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996876, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:07:56.792400 24701 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:07:56.803320 24701 webserver.cc:460] Webserver started at http://127.12.2.65:44283/ using document root <none> and password file <none>
I20260620 23:07:56.804122 24701 fs_manager.cc:359] Metadata directory not provided
I20260620 23:07:56.804250 24701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:07:56.804543 24701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:07:56 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996876, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:56.824400 24701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "ab1f290bbac84a23acb87d5e71e773e9"
format_stamp: "Formatted at 2026-06-20 23:07:56 on dist-test-slave-7g8g"
server_key: "7602a640111c4542c2dd9a766213a5c4"
server_key_iv: "a10422b0e4c1d2c1b89070605bb7be25"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:56.825755 24701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "ab1f290bbac84a23acb87d5e71e773e9"
format_stamp: "Formatted at 2026-06-20 23:07:56 on dist-test-slave-7g8g"
server_key: "7602a640111c4542c2dd9a766213a5c4"
server_key_iv: "a10422b0e4c1d2c1b89070605bb7be25"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:56.848994 24701 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.015s sys 0.007s
I20260620 23:07:56.887449 24721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:56.889468 24701 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.008s sys 0.011s
I20260620 23:07:56.889737 24701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "ab1f290bbac84a23acb87d5e71e773e9"
format_stamp: "Formatted at 2026-06-20 23:07:56 on dist-test-slave-7g8g"
server_key: "7602a640111c4542c2dd9a766213a5c4"
server_key_iv: "a10422b0e4c1d2c1b89070605bb7be25"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:56.890192 24701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:07:56.926820 24701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:07:56.930722 24701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:07:56.931039 24701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:07:56.931951 24701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:07:56.933306 24701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:07:56.933380 24701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:56.933497 24701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:07:56.933547 24701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:57.020499 24701 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:40717
I20260620 23:07:57.020502 24833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:40717 every 8 connection(s)
I20260620 23:07:57.022321 24701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:07:57.022699 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 24701
I20260620 23:07:57.022900 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:07:57.023820 12297 external_mini_cluster.cc:1428] Setting key 5c288c6a3b366f68e8f7b05c48398fee
Jun 20 23:07:57 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996876, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
I20260620 23:07:57.069375 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:57.032410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:57711 (local address 127.12.2.126:37163)
0620 23:07:57.039535 (+ 7125us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:57.039542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:57.042975 (+ 3433us) server_negotiation.cc:407] Connection header received
0620 23:07:57.043063 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:57.043070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:57.043173 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:57.043270 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:57.045266 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:57.046001 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:57.055176 (+ 9175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:57.055884 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:57.056023 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:57.062394 (+ 6371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:57.062401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:57.062406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:57.062451 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:57.065650 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:57.066640 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:57.066645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:57.066648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:57.066809 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:57.067375 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:57.067380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:57.067382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:57.067720 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:57.067842 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:57.068793 (+ 951us) server_negotiation.cc:299] Negotiation successful
0620 23:07:57.069011 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6981,"thread_start_us":6880,"threads_started":1}
I20260620 23:07:57.071798 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:57.031811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37163 (local address 127.12.2.65:57711)
0620 23:07:57.040472 (+ 8661us) negotiation.cc:107] Waiting for socket to connect
0620 23:07:57.040533 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:07:57.042740 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:07:57.043804 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:07:57.043816 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:07:57.044431 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:07:57.045100 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:57.045118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:57.046145 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:57.046149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:57.047621 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:57.047649 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:57.056169 (+ 8520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:57.056178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:57.056489 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:57.057338 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:07:57.057372 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:07:57.059834 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:07:57.065826 (+ 5992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:57.065835 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:57.065855 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:57.066494 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:57.066970 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:57.066974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:57.066979 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:57.067248 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:57.067943 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:07:57.067952 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:07:57.068373 (+ 421us) client_negotiation.cc:769] Sending connection context
0620 23:07:57.069874 (+ 1501us) client_negotiation.cc:240] Negotiation successful
0620 23:07:57.070427 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8377,"thread_start_us":8151,"threads_started":1}
I20260620 23:07:57.073745 24834 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37163
I20260620 23:07:57.074041 24834 heartbeater.cc:464] Registering TS with master...
I20260620 23:07:57.074679 24834 heartbeater.cc:511] Master 127.12.2.126:37163 requested a full tablet report, sending...
I20260620 23:07:57.077087 24621 ts_manager.cc:194] Registered new tserver with Master: ab1f290bbac84a23acb87d5e71e773e9 (127.12.2.65:40717)
I20260620 23:07:57.078661 24621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:57711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:07:57.098462 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:50887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37163
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:07:57 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996876, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
Jun 20 23:07:57 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996876, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
W20260620 23:07:57.607842 24840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:07:57.608390 24840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:07:57.608594 24840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:07:57.619726 24840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:07:57.619935 24840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:07:57.620182 24840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:07:57.635119 24840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.2.84:50887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:07:57.637042 24840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:07:57.638784 24840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:07:57.661315 24847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:57.671650 24848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:57.685195 24851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:07:57.693213 24840 server_base.cc:1034] running on GCE node
I20260620 23:07:57.694175 24840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:07:57.695998 24840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:07:57.699432 24840 hybrid_clock.cc:648] HybridClock initialized: now 1781996877699294 us; error 135 us; skew 500 ppm
I20260620 23:07:57.699841 24840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:07:57 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996877, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:07:57.704362 24840 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:07:57.716377 24840 webserver.cc:460] Webserver started at http://127.12.2.66:35245/ using document root <none> and password file <none>
I20260620 23:07:57.717445 24840 fs_manager.cc:359] Metadata directory not provided
I20260620 23:07:57.717654 24840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:07:57.718117 24840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:07:57 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996877, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:57.759745 24840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "06dc406fe0214c96b029e8f04283e76d"
format_stamp: "Formatted at 2026-06-20 23:07:57 on dist-test-slave-7g8g"
server_key: "cd84c221995f908b3911f8166ae9e221"
server_key_iv: "e7cab0571267d242d130f4f1faae4452"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:57.761056 24840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "06dc406fe0214c96b029e8f04283e76d"
format_stamp: "Formatted at 2026-06-20 23:07:57 on dist-test-slave-7g8g"
server_key: "cd84c221995f908b3911f8166ae9e221"
server_key_iv: "e7cab0571267d242d130f4f1faae4452"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:57.781188 24840 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.007s sys 0.000s
I20260620 23:07:57.811964 24860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:57.813832 24840 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.001s sys 0.005s
I20260620 23:07:57.814020 24840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "06dc406fe0214c96b029e8f04283e76d"
format_stamp: "Formatted at 2026-06-20 23:07:57 on dist-test-slave-7g8g"
server_key: "cd84c221995f908b3911f8166ae9e221"
server_key_iv: "e7cab0571267d242d130f4f1faae4452"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:57.814252 24840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:07:57.896562 24840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:07:57.901832 24840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:07:57.902205 24840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:07:57.915764 24840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:07:57.923992 24840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:07:57.924187 24840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 23:07:57.924355 24840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:07:57.924443 24840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:58.054919 24840 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:39077
I20260620 23:07:58.054952 24976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:39077 every 8 connection(s)
I20260620 23:07:58.056263 24840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:07:58.057803 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 24840
I20260620 23:07:58.057964 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:07:58.058524 12297 external_mini_cluster.cc:1428] Setting key e7aee80bb375baa1133bd23c40c3c80b
Jun 20 23:07:58 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996877, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:07:58.081955 24834 heartbeater.cc:503] Master 127.12.2.126:37163 was elected leader, sending a full tablet report...
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
I20260620 23:07:58.088225 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:58.060394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:58207 (local address 127.12.2.126:37163)
0620 23:07:58.070822 (+ 10428us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:58.070829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:58.070849 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:07:58.070908 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:58.070913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:58.071021 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:58.071131 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:58.073126 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.074170 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.076416 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.077004 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.077126 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:58.081185 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:58.081192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:58.081197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:58.081241 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:58.084866 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:58.085747 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:58.085752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:58.085756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:58.085909 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:58.086357 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:58.086362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:58.086366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:58.086729 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:58.086866 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:58.087553 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 23:07:58.087851 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10306,"thread_start_us":10197,"threads_started":1}
I20260620 23:07:58.089190 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:58.059804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37163 (local address 127.12.2.66:58207)
0620 23:07:58.067549 (+ 7745us) negotiation.cc:107] Waiting for socket to connect
0620 23:07:58.067598 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:07:58.070060 (+ 2462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:07:58.071554 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:07:58.071567 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:07:58.072164 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:07:58.072914 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:58.072938 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.074347 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.074351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:58.076242 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:58.076265 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.077420 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.077427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:58.077681 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:58.078402 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:07:58.078430 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:07:58.080934 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:07:58.085011 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:58.085020 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:58.085039 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:58.085614 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:58.086005 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:58.086008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:58.086011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:58.086236 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:58.086842 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:07:58.086848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:07:58.087211 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 23:07:58.087434 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 23:07:58.087840 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7447,"thread_start_us":7216,"threads_started":1}
I20260620 23:07:58.093113 24977 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37163
I20260620 23:07:58.093461 24977 heartbeater.cc:464] Registering TS with master...
I20260620 23:07:58.094322 24977 heartbeater.cc:511] Master 127.12.2.126:37163 requested a full tablet report, sending...
I20260620 23:07:58.095960 24621 ts_manager.cc:194] Registered new tserver with Master: 06dc406fe0214c96b029e8f04283e76d (127.12.2.66:39077)
I20260620 23:07:58.097014 24621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:58207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:07:58.114094 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:50887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37163
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:07:58.418820 24983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:07:58.419126 24983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:07:58.419221 24983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:07:58.426376 24983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:07:58.426478 24983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:07:58.426594 24983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:07:58.436173 24983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.2.84:50887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:37163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:07:58.437371 24983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:07:58.438555 24983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:07:58.450521 24991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:58.450569 24988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:58.450572 24989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:07:58.451694 24983 server_base.cc:1034] running on GCE node
I20260620 23:07:58.452256 24983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:07:58.453135 24983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:07:58.454329 24983 hybrid_clock.cc:648] HybridClock initialized: now 1781996878454279 us; error 37 us; skew 500 ppm
I20260620 23:07:58.454620 24983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:07:58 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996878, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:07:58.457470 24983 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:07:58.458813 24983 webserver.cc:460] Webserver started at http://127.12.2.67:38955/ using document root <none> and password file <none>
I20260620 23:07:58.459492 24983 fs_manager.cc:359] Metadata directory not provided
I20260620 23:07:58.459614 24983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:07:58.459883 24983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:07:58 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996878, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:58.473124 24983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "9a27763a9a3b484ea5abd1b53d30cfb2"
format_stamp: "Formatted at 2026-06-20 23:07:58 on dist-test-slave-7g8g"
server_key: "48f5229b1113497650c5d968063d295c"
server_key_iv: "e1432ca4b5ee9053aaf0d94210332457"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:58.473887 24983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "9a27763a9a3b484ea5abd1b53d30cfb2"
format_stamp: "Formatted at 2026-06-20 23:07:58 on dist-test-slave-7g8g"
server_key: "48f5229b1113497650c5d968063d295c"
server_key_iv: "e1432ca4b5ee9053aaf0d94210332457"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:58.479158 24983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:07:58.496374 24998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:58.497864 24983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:07:58.498014 24983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "9a27763a9a3b484ea5abd1b53d30cfb2"
format_stamp: "Formatted at 2026-06-20 23:07:58 on dist-test-slave-7g8g"
server_key: "48f5229b1113497650c5d968063d295c"
server_key_iv: "e1432ca4b5ee9053aaf0d94210332457"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:58.498215 24983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:07:58.515738 24983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:07:58.519181 24983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:07:58.519434 24983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:07:58.520375 24983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:07:58.521646 24983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:07:58.521714 24983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:58.521821 24983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:07:58.521860 24983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:58.550239 24983 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:32777
I20260620 23:07:58.550257 25110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:32777 every 8 connection(s)
I20260620 23:07:58.551525 24983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:07:58.553882 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 24983
I20260620 23:07:58.554052 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:07:58.554486 12297 external_mini_cluster.cc:1428] Setting key 62df08b13b39635c7aeff3422c170376
Jun 20 23:07:58 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996878, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:07:58.569819 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:58.554055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:57773 (local address 127.12.2.126:37163)
0620 23:07:58.554243 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:58.554249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:58.555401 (+ 1152us) server_negotiation.cc:407] Connection header received
0620 23:07:58.557064 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:58.557070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:58.557171 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:58.557246 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:58.558531 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.559299 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.560773 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.561155 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.561249 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:58.564266 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:58.564272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:58.564275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:58.564309 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:58.566688 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:58.567390 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:58.567395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:58.567397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:58.567523 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:58.567952 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:58.567957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:58.567960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:58.568319 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:58.568453 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:58.569160 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 23:07:58.569498 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:07:58.570614 25113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:58.554491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37163 (local address 127.12.2.67:57773)
0620 23:07:58.555272 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 23:07:58.555315 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:07:58.556850 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:07:58.557468 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:07:58.557476 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:07:58.557864 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:07:58.558400 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:58.558416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.559440 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.559444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:58.560669 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:07:58.560680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.561314 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.561318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:07:58.561510 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:58.562115 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:07:58.562136 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:07:58.564054 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:07:58.566821 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:58.566830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:58.566845 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:58.567282 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:58.567619 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:07:58.567624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:07:58.567628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:07:58.567851 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:07:58.568436 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:07:58.568443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:07:58.568784 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 23:07:58.569038 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 23:07:58.569525 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":336,"threads_started":1}
I20260620 23:07:58.572049 25111 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37163
I20260620 23:07:58.572317 25111 heartbeater.cc:464] Registering TS with master...
I20260620 23:07:58.572986 25111 heartbeater.cc:511] Master 127.12.2.126:37163 requested a full tablet report, sending...
I20260620 23:07:58.574296 24621 ts_manager.cc:194] Registered new tserver with Master: 9a27763a9a3b484ea5abd1b53d30cfb2 (127.12.2.67:32777)
I20260620 23:07:58.575131 24621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:57773
I20260620 23:07:58.581715 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:07:58.583343 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 24701
Jun 20 23:07:58 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:07:58.598773 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:58.584131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.12.2.65:40717)
0620 23:07:58.584729 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:58.584736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:58.584761 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:07:58.584814 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:58.584819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:58.585001 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:58.585126 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:58.586521 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.587595 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.588940 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.589464 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.589555 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:58.593407 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:58.593419 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:58.593432 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:58.593469 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:58.596252 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:58.596889 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:58.596897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:58.596899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:58.597099 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:58.597428 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:58.597431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:58.597434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:58.597806 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:58.598054 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:58.598249 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 23:07:58.598450 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
I20260620 23:07:58.599982 24829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:07:58.682366 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 24701
I20260620 23:07:58.694387 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 24840
Jun 20 23:07:58 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:07:58.710129 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:58.695293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58454 (local address 127.12.2.66:39077)
0620 23:07:58.695805 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:58.695812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:58.695836 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:07:58.695910 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:58.695916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:58.696137 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:58.696268 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:58.697428 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.698453 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.699890 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.700453 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.700557 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:58.704638 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:58.704649 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:58.704663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:58.704696 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:58.707739 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:58.708329 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:58.708336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:58.708339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:58.708466 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:58.708761 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:58.708764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:58.708766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:58.709110 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:58.709347 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:58.709636 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 23:07:58.709832 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":258,"threads_started":1}
I20260620 23:07:58.711444 24972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:07:58.790616 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 24840
I20260620 23:07:58.802552 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 24983
Jun 20 23:07:58 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:07:58.818464 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:07:58.803552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.12.2.67:32777)
0620 23:07:58.804178 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 23:07:58.804185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:07:58.804213 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:07:58.804271 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:07:58.804277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:07:58.804486 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:07:58.804619 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:07:58.805941 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.807117 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.808328 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:07:58.808861 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:07:58.808954 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:07:58.813102 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:07:58.813112 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:07:58.813127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:07:58.813158 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:07:58.816001 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:58.816651 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:58.816666 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:58.816668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:58.816794 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:07:58.817087 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:07:58.817090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:07:58.817092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:07:58.817427 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:07:58.817693 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:07:58.817892 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 23:07:58.818129 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":346,"threads_started":1}
I20260620 23:07:58.819751 25106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:07:58.899506 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 24983
I20260620 23:07:58.911808 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 24589
I20260620 23:07:58.912987 24651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:07:59.015627 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 24589
I20260620 23:07:59.035666 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:37163
--webserver_interface=127.12.2.126
--webserver_port=39125
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:37163
--ranger_config_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:50887/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.2.84:50887/kms with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:07:59.345049 25122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:07:59.345368 25122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:07:59.345517 25122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:07:59.352969 25122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:07:59.353094 25122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:07:59.353175 25122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:07:59.353256 25122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:07:59.353325 25122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:07:59.363734 25122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.2.84:50887/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:37163
--ranger_config_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:37163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=39125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:07:59.365038 25122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:07:59.366430 25122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:07:59.378258 25127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:59.378187 25128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:59.378441 25130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:07:59.379601 25122 server_base.cc:1034] running on GCE node
I20260620 23:07:59.380338 25122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:07:59.381693 25122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:07:59.382920 25122 hybrid_clock.cc:648] HybridClock initialized: now 1781996879382863 us; error 39 us; skew 500 ppm
I20260620 23:07:59.383311 25122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:07:59 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996879, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:07:59.386129 25122 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:07:59.387492 25122 webserver.cc:460] Webserver started at http://127.12.2.126:39125/ using document root <none> and password file <none>
I20260620 23:07:59.388120 25122 fs_manager.cc:359] Metadata directory not provided
I20260620 23:07:59.388233 25122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:07:59 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996879, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:07:59.468616 25122 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:07:59.471630 25138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:07:59.473069 25122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:07:59.473212 25122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "1fdf57ebc400408b977fbce4aec70cf3"
format_stamp: "Formatted at 2026-06-20 23:07:55 on dist-test-slave-7g8g"
server_key: "fcbbcb30c53b43e14ce1e32aa05e1a6b"
server_key_iv: "331ae177ee25a10f4ad24bca03d7a068"
server_key_version: "kuduclusterkey@0"
I20260620 23:07:59.474151 25122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:07:59.497200 25122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:07:59.501159 25122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:07:59.501423 25122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:07:59.518141 25122 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:37163
I20260620 23:07:59.518174 25189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:37163 every 8 connection(s)
I20260620 23:07:59.519626 25122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:07:59.526386 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 25122
I20260620 23:07:59.526733 25190 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:07:59.527261 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:50887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:40717
--local_ip_for_outbound_sockets=127.12.2.65
--tserver_master_addrs=127.12.2.126:37163
--webserver_port=44283
--webserver_interface=127.12.2.65
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.2.84:50887/kms with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:07:59.530648 25190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3: Bootstrap starting.
I20260620 23:07:59.549285 25190 log.cc:824] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:07:59.558372 25190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:07:59.558841 25190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3: Bootstrap complete.
I20260620 23:07:59.567317 25190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } }
I20260620 23:07:59.568130 25190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdf57ebc400408b977fbce4aec70cf3, State: Initialized, Role: FOLLOWER
I20260620 23:07:59.568691 25190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } }
I20260620 23:07:59.568902 25190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:07:59.569001 25190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:07:59.569155 25190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:07:59.573082 25190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } }
I20260620 23:07:59.573519 25190 leader_election.cc:302] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fdf57ebc400408b977fbce4aec70cf3; no voters:
I20260620 23:07:59.574087 25190 leader_election.cc:288] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:07:59.574227 25194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:07:59.575179 25194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [term 2 LEADER]: Becoming Leader. State: Replica: 1fdf57ebc400408b977fbce4aec70cf3, State: Running, Role: LEADER
I20260620 23:07:59.575762 25194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } }
I20260620 23:07:59.576159 25190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:07:59.578235 25195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1fdf57ebc400408b977fbce4aec70cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } } }
I20260620 23:07:59.578601 25195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:07:59.583613 25196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fdf57ebc400408b977fbce4aec70cf3. Latest consensus state: current_term: 2 leader_uuid: "1fdf57ebc400408b977fbce4aec70cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdf57ebc400408b977fbce4aec70cf3" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 37163 } } }
I20260620 23:07:59.583930 25196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:07:59.586434 25199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:07:59.593179 25199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:07:59.594514 25199 catalog_manager.cc:1259] Loaded cluster ID: d9e494a135ac498db8c12498c82c9b26
I20260620 23:07:59.594604 25199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:07:59.595527 25190 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:07:59.598628 25199 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:07:59.600059 25199 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3: Loaded TSK: 0
I20260620 23:07:59.601562 25199 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:07:59.933254 25192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:07:59.933652 25192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:07:59.933851 25192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:07:59.944993 25192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:07:59.945180 25192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:07:59.945392 25192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:07:59.960328 25192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.2.84:50887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:40717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=44283
--tserver_master_addrs=127.12.2.126:37163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:07:59.962093 25192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:07:59.963832 25192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:07:59.988742 25227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:07:59.992204 25192 server_base.cc:1034] running on GCE node
W20260620 23:07:59.992273 25228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:07:59.992844 25231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:07:59.993695 25192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:07:59.998675 25192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:07:59.999985 25192 hybrid_clock.cc:648] HybridClock initialized: now 1781996879999832 us; error 152 us; skew 500 ppm
I20260620 23:08:00.000474 25192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:00 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996880, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:00.005024 25192 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:08:00.012713 25192 webserver.cc:460] Webserver started at http://127.12.2.65:44283/ using document root <none> and password file <none>
I20260620 23:08:00.013767 25192 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:00.013947 25192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:08:00 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996880, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:08:00.063428 25192 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.006s sys 0.005s
I20260620 23:08:00.079650 25238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:00.084694 25192 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.012s sys 0.003s
I20260620 23:08:00.084936 25192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "ab1f290bbac84a23acb87d5e71e773e9"
format_stamp: "Formatted at 2026-06-20 23:07:56 on dist-test-slave-7g8g"
server_key: "7602a640111c4542c2dd9a766213a5c4"
server_key_iv: "a10422b0e4c1d2c1b89070605bb7be25"
server_key_version: "kuduclusterkey@0"
I20260620 23:08:00.086643 25192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:00.157626 25192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:00.162963 25192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:00.163363 25192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:00.166203 25192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:00.167747 25192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:00.167843 25192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:00.168004 25192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:00.168074 25192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:00.225678 25192 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:40717
I20260620 23:08:00.225739 25351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:40717 every 8 connection(s)
I20260620 23:08:00.227195 25192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:08:00.231683 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 25192
I20260620 23:08:00.232316 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:50887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:39077
--local_ip_for_outbound_sockets=127.12.2.66
--tserver_master_addrs=127.12.2.126:37163
--webserver_port=35245
--webserver_interface=127.12.2.66
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.2.84:50887/kms with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:08:00 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996880, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:00.262351 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:00.230843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37163 (local address 127.12.2.65:55775)
0620 23:08:00.240345 (+ 9502us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:00.240403 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:00.242697 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:00.245327 (+ 2630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:00.245339 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:00.245954 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:00.246756 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:00.246781 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:00.249218 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:00.249224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:00.250505 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:00.250517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:00.251637 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:00.251644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:00.251892 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:00.252498 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:00.252520 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:00.254675 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:00.257965 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:00.257974 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:00.257990 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:00.258378 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:00.258787 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:00.258792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:00.258796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:00.259013 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:00.259734 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:00.259741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:00.260046 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 23:08:00.260280 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:08:00.260825 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9166,"thread_start_us":12719,"threads_started":1}
I20260620 23:08:00.263167 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:00.230670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:55775 (local address 127.12.2.126:37163)
0620 23:08:00.235292 (+ 4622us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:00.235313 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:00.243223 (+ 7910us) server_negotiation.cc:407] Connection header received
0620 23:08:00.243732 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:00.243750 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:00.244262 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:00.244689 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:00.247005 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:00.248987 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:00.250651 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:00.251399 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:00.251583 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:00.254901 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:00.254908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:00.254923 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:00.254994 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:00.257829 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:00.258508 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:00.258515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:00.258520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:00.258673 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:00.259126 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:00.259130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:00.259132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:00.259596 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:00.259771 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:00.260509 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 23:08:00.260865 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4333,"thread_start_us":269,"threads_started":1}
I20260620 23:08:00.266016 25352 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37163
I20260620 23:08:00.266320 25352 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:00.267253 25352 heartbeater.cc:511] Master 127.12.2.126:37163 requested a full tablet report, sending...
W20260620 23:08:00.270503 25352 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:37163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:08:00.270730 25352 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:00.271016 25352 heartbeater.cc:511] Master 127.12.2.126:37163 requested a full tablet report, sending...
I20260620 23:08:00.272375 25352 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:00.272724 25352 heartbeater.cc:511] Master 127.12.2.126:37163 requested a full tablet report, sending...
W20260620 23:08:00.273813 25352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:08:00.679808 25356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:00.680172 25356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:00.680371 25356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:00.691606 25356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:00.691733 25356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:00.691955 25356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:08:00.707130 25356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.2.84:50887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:39077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=35245
--tserver_master_addrs=127.12.2.126:37163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:00.708747 25356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:00.710399 25356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:00.745256 25371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:00.745848 25370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:00.753685 25356 server_base.cc:1034] running on GCE node
W20260620 23:08:00.754171 25373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:00.755031 25356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:00.764289 25356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:00.765578 25356 hybrid_clock.cc:648] HybridClock initialized: now 1781996880765495 us; error 78 us; skew 500 ppm
I20260620 23:08:00.766065 25356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:00 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996880, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:00.774091 25356 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:08:00.776546 25356 webserver.cc:460] Webserver started at http://127.12.2.66:35245/ using document root <none> and password file <none>
I20260620 23:08:00.777513 25356 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:00.777732 25356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:08:00 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996880, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:08:00.807943 25356 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.001s
I20260620 23:08:00.817013 25383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:00.819010 25356 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.007s
I20260620 23:08:00.819226 25356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "06dc406fe0214c96b029e8f04283e76d"
format_stamp: "Formatted at 2026-06-20 23:07:57 on dist-test-slave-7g8g"
server_key: "cd84c221995f908b3911f8166ae9e221"
server_key_iv: "e7cab0571267d242d130f4f1faae4452"
server_key_version: "kuduclusterkey@0"
I20260620 23:08:00.820669 25356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 23:08:00 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996880, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:00.847914 25356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:00.852890 25356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:00.853267 25356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:00.854497 25356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:00.862551 25356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:00.862757 25356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 23:08:00.862943 25356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:00.863065 25356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:00.914172 25356 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:39077
I20260620 23:08:00.914247 25496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:39077 every 8 connection(s)
I20260620 23:08:00.916006 25356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:08:00.923370 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 25356
I20260620 23:08:00.924109 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.2.84:50887/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:32777
--local_ip_for_outbound_sockets=127.12.2.67
--tserver_master_addrs=127.12.2.126:37163
--webserver_port=38955
--webserver_interface=127.12.2.67
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.2.84:50887/kms with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:08:00 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996880, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:00.950865 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:00.919874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37163 (local address 127.12.2.66:48975)
0620 23:08:00.931522 (+ 11648us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:00.931583 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:00.933996 (+ 2413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:00.934845 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:00.934856 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:00.935498 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:00.936200 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:00.936227 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:00.937533 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:00.937537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:00.938791 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:00.938813 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:00.940051 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:00.940056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:00.940247 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:00.940854 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:00.940876 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:00.942821 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:00.946103 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:00.946112 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:00.946130 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:00.946722 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:00.947198 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:00.947203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:00.947207 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:00.947442 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:00.948189 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:00.948198 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:00.948600 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 23:08:00.948915 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 23:08:00.949480 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11322,"thread_start_us":11099,"threads_started":1}
I20260620 23:08:00.951951 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:00.919218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:48975 (local address 127.12.2.126:37163)
0620 23:08:00.921190 (+ 1972us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:00.921196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:00.932611 (+ 11415us) server_negotiation.cc:407] Connection header received
0620 23:08:00.934366 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:00.934371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:00.934445 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:00.934552 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:00.936419 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:00.937320 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:00.938967 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:00.939614 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:00.939766 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:00.943042 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:00.943047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:00.943050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:00.943081 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:00.945941 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:00.946867 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:00.946873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:00.946876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:00.947051 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:00.947593 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:00.947598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:00.947601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:00.947962 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:00.948090 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:00.951362 (+ 3272us) server_negotiation.cc:299] Negotiation successful
0620 23:08:00.951590 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1839,"thread_start_us":1722,"threads_started":1}
I20260620 23:08:00.954164 25497 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37163
I20260620 23:08:00.954530 25497 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:00.955451 25497 heartbeater.cc:511] Master 127.12.2.126:37163 requested a full tablet report, sending...
I20260620 23:08:00.957486 25155 ts_manager.cc:194] Registered new tserver with Master: 06dc406fe0214c96b029e8f04283e76d (127.12.2.66:39077)
I20260620 23:08:00.959703 25155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:48975
Jun 20 23:08:01 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996880, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:08:01.273990 25352 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:01.274430 25352 heartbeater.cc:511] Master 127.12.2.126:37163 requested a full tablet report, sending...
I20260620 23:08:01.275723 25155 ts_manager.cc:194] Registered new tserver with Master: ab1f290bbac84a23acb87d5e71e773e9 (127.12.2.65:40717)
I20260620 23:08:01.276914 25155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:55775
W20260620 23:08:01.349350 25501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:01.349834 25501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:01.350023 25501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:01.362854 25501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:01.363108 25501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:01.363343 25501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:08:01.378593 25501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.2.84:50887/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:32777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=38955
--tserver_master_addrs=127.12.2.126:37163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:01.380456 25501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:01.382133 25501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:01.424659 25507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:01.428479 25510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:01.428931 25501 server_base.cc:1034] running on GCE node
W20260620 23:08:01.429613 25506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:01.430493 25501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:01.437574 25501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:01.438838 25501 hybrid_clock.cc:648] HybridClock initialized: now 1781996881438772 us; error 66 us; skew 500 ppm
I20260620 23:08:01.439311 25501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:01 dist-test-slave-7g8g krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996881, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:01.443387 25501 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:08:01.463864 25501 webserver.cc:460] Webserver started at http://127.12.2.67:38955/ using document root <none> and password file <none>
I20260620 23:08:01.465077 25501 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:01.465353 25501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:08:01 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996881, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:08:01.498492 25501 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:08:01.518311 25517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:01.522204 25501 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.007s sys 0.012s
I20260620 23:08:01.523111 25501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "9a27763a9a3b484ea5abd1b53d30cfb2"
format_stamp: "Formatted at 2026-06-20 23:07:58 on dist-test-slave-7g8g"
server_key: "48f5229b1113497650c5d968063d295c"
server_key_iv: "e1432ca4b5ee9053aaf0d94210332457"
server_key_version: "kuduclusterkey@0"
I20260620 23:08:01.526438 25501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:01.594687 25501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:01.599841 25501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:01.600205 25501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:01.607535 25501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:01.615592 25501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:01.615790 25501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:01.616042 25501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:01.616169 25501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:08:01 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996880, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for HTTP/127.12.2.84@KRBTEST.COM
I20260620 23:08:01.883950 25501 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:32777
I20260620 23:08:01.883992 25630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:32777 every 8 connection(s)
I20260620 23:08:01.885460 25501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:08:01.893714 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 25501
Jun 20 23:08:01 dist-test-slave-7g8g krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996881, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:01.935429 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:01.889681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:48701 (local address 127.12.2.126:37163)
0620 23:08:01.910036 (+ 20355us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:01.910043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:01.912528 (+ 2485us) server_negotiation.cc:407] Connection header received
0620 23:08:01.914727 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:01.914732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:01.914816 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:01.914922 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:01.917073 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:01.918258 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:01.920958 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:01.921939 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:01.922039 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:01.928213 (+ 6174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:01.928220 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:01.928225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:01.928266 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:01.931689 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:01.932644 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:01.932650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:01.932653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:01.932792 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:01.933333 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:01.933338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:01.933341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:01.933688 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:01.933817 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:01.934796 (+ 979us) server_negotiation.cc:299] Negotiation successful
0620 23:08:01.935030 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20201,"thread_start_us":21416,"threads_started":1}
I20260620 23:08:01.937654 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:01.894911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.12.2.126:37163)
0620 23:08:01.911364 (+ 16453us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:01.911370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:01.915102 (+ 3732us) server_negotiation.cc:407] Connection header received
0620 23:08:01.920365 (+ 5263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:01.920371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:01.920447 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:01.920531 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:01.922576 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:01.923773 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:01.926599 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:01.927296 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:01.927410 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:01.931318 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:01.931341 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:01.931346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:01.931405 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:01.934623 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:01.935465 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:01.935470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:01.935473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:01.935606 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:01.936034 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:01.936038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:01.936041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:01.936338 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:01.936570 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:01.937127 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 23:08:01.937340 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16335,"thread_start_us":15109,"threads_started":1}
I20260620 23:08:01.937956 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:01.889011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:37163 (local address 127.12.2.67:48701)
0620 23:08:01.911930 (+ 22919us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:01.911992 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:01.914262 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:01.915425 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:01.915437 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:01.916051 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:01.916874 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:01.916899 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:01.918447 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:01.918452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:01.920794 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:01.920820 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:01.922415 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:01.922422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:01.924159 (+ 1737us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:01.925017 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:01.925049 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:01.927936 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:01.931849 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:01.931859 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:01.931877 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:01.932511 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:01.932938 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:01.932943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:01.932946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:01.933197 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:01.933912 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:01.933920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:01.934380 (+ 460us) client_negotiation.cc:769] Sending connection context
0620 23:08:01.935957 (+ 1577us) client_negotiation.cc:240] Negotiation successful
0620 23:08:01.936538 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22622,"thread_start_us":22393,"threads_started":1}
I20260620 23:08:01.940593 25631 heartbeater.cc:347] Connected to a master server at 127.12.2.126:37163
I20260620 23:08:01.940907 25631 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:01.941779 25631 heartbeater.cc:511] Master 127.12.2.126:37163 requested a full tablet report, sending...
I20260620 23:08:01.943957 25155 ts_manager.cc:194] Registered new tserver with Master: 9a27763a9a3b484ea5abd1b53d30cfb2 (127.12.2.67:32777)
I20260620 23:08:01.945065 25155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:48701
I20260620 23:08:01.956279 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:08:01.964372 25497 heartbeater.cc:503] Master 127.12.2.126:37163 was elected leader, sending a full tablet report...
I20260620 23:08:01.974442 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:01.959339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.12.2.126:37163)
0620 23:08:01.959521 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:01.959527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:01.960339 (+ 812us) server_negotiation.cc:407] Connection header received
0620 23:08:01.960579 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:01.960583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:01.960691 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:01.960798 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:01.962780 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:01.963582 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:01.965563 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:01.966245 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:01.966378 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:01.968684 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:01.968691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:01.968696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:01.968735 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:01.971987 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:01.972615 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:01.972621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:01.972624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:01.972776 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:01.973069 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:01.973074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:01.973077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:01.973430 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:01.973661 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:01.973887 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 23:08:01.974107 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:08:01.979696 25155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:08:01.987013 25155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:08:02.025723 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.004077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:39077 (local address 127.0.0.1:58462)
0620 23:08:02.014270 (+ 10193us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.014295 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:02.014401 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:02.016906 (+ 2505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:02.016912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:02.016939 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:02.017333 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:02.017345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.019568 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.019574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:02.021777 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:02.021794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.025036 (+ 3242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.025046 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:02.025208 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.025239 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:08:02.025303 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:08:02.025418 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10080,"thread_start_us":9604,"threads_started":1}
I20260620 23:08:02.025894 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.002111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.12.2.65:40717)
0620 23:08:02.014991 (+ 12880us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:02.015001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:02.015034 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:08:02.015133 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:02.015142 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:02.015414 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:02.015609 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:02.016324 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.019002 (+ 2678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.021711 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.024124 (+ 2413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.024262 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.024491 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:02.025398 (+ 907us) server_negotiation.cc:299] Negotiation successful
0620 23:08:02.025615 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12674,"thread_start_us":12495,"threads_started":1}
I20260620 23:08:02.026144 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.001647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:40717 (local address 127.0.0.1:53226)
0620 23:08:02.013951 (+ 12304us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.013967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:02.014078 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:02.015652 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:02.015659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:02.015689 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:02.016108 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:02.016122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.019174 (+ 3052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.019179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:02.021546 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:02.021560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.024323 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.024328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:02.024517 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.024581 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 23:08:02.024705 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 23:08:02.024906 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12187,"mutex_wait_us":3,"thread_start_us":11819,"threads_started":1}
I20260620 23:08:02.027491 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.004719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58462 (local address 127.12.2.66:39077)
0620 23:08:02.016306 (+ 11587us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:02.016315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:02.016342 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:08:02.016437 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:02.016445 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:02.016691 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:02.017537 (+ 846us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:02.017781 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.019393 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.021998 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.023577 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.026754 (+ 3177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.027000 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:02.027064 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 23:08:02.027218 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11433,"thread_start_us":11287,"threads_started":1}
I20260620 23:08:02.031179 25431 tablet_service.cc:1467] Processing CreateTablet for tablet de686616199440b19230adebc909ea38 (DEFAULT_TABLE table=test-table [id=ded3744606b3450aa68f76189a1b6a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:02.031172 25287 tablet_service.cc:1467] Processing CreateTablet for tablet de686616199440b19230adebc909ea38 (DEFAULT_TABLE table=test-table [id=ded3744606b3450aa68f76189a1b6a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:02.032735 25287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de686616199440b19230adebc909ea38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:02.032752 25431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de686616199440b19230adebc909ea38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:02.037969 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.001490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:32777 (local address 127.0.0.1:56888)
0620 23:08:02.013057 (+ 11567us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.013102 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:02.013281 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:02.024619 (+ 11338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:02.024624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:02.024649 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:02.025030 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:02.025041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.027026 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.027031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:02.028888 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:02.028902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.037038 (+ 8136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.037046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:02.037256 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.037313 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 23:08:02.037393 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:08:02.037546 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11289,"thread_start_us":12665,"threads_started":1}
I20260620 23:08:02.040908 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.002126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.12.2.67:32777)
0620 23:08:02.024041 (+ 21915us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:02.024049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:02.024072 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:08:02.024150 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:02.024155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:02.024381 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:02.024616 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:02.025256 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.026857 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.035168 (+ 8311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.036756 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.039996 (+ 3240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.040285 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:02.040363 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 23:08:02.040538 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21753,"thread_start_us":21646,"threads_started":1}
I20260620 23:08:02.044902 25566 tablet_service.cc:1467] Processing CreateTablet for tablet de686616199440b19230adebc909ea38 (DEFAULT_TABLE table=test-table [id=ded3744606b3450aa68f76189a1b6a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:02.046556 25566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de686616199440b19230adebc909ea38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:02.054143 25659 tablet_bootstrap.cc:492] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9: Bootstrap starting.
I20260620 23:08:02.057451 25660 tablet_bootstrap.cc:492] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: Bootstrap starting.
I20260620 23:08:02.060534 25660 tablet_bootstrap.cc:654] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:02.061769 25660 log.cc:824] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:02.063601 25659 tablet_bootstrap.cc:654] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:02.064965 25659 log.cc:824] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:02.068684 25661 tablet_bootstrap.cc:492] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: Bootstrap starting.
I20260620 23:08:02.071004 25660 tablet_bootstrap.cc:492] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: No bootstrap required, opened a new log
I20260620 23:08:02.071291 25660 ts_tablet_manager.cc:1397] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260620 23:08:02.071914 25659 tablet_bootstrap.cc:492] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9: No bootstrap required, opened a new log
I20260620 23:08:02.072158 25659 ts_tablet_manager.cc:1397] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.001s
I20260620 23:08:02.078176 25661 tablet_bootstrap.cc:654] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:02.079648 25661 log.cc:824] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:02.081668 25660 raft_consensus.cc:348] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32777 } } peers { permanent_uuid: "ab1f290bbac84a23acb87d5e71e773e9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 40717 } } peers { permanent_uuid: "06dc406fe0214c96b029e8f04283e76d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 39077 } }
I20260620 23:08:02.082234 25660 raft_consensus.cc:374] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:02.082386 25660 raft_consensus.cc:729] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06dc406fe0214c96b029e8f04283e76d, State: Initialized, Role: FOLLOWER
I20260620 23:08:02.082297 25659 raft_consensus.cc:348] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32777 } } peers { permanent_uuid: "ab1f290bbac84a23acb87d5e71e773e9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 40717 } } peers { permanent_uuid: "06dc406fe0214c96b029e8f04283e76d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 39077 } }
I20260620 23:08:02.082866 25659 raft_consensus.cc:374] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:02.083022 25659 raft_consensus.cc:729] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1f290bbac84a23acb87d5e71e773e9, State: Initialized, Role: FOLLOWER
I20260620 23:08:02.083199 25660 consensus_queue.cc:260] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32777 } } peers { permanent_uuid: "ab1f290bbac84a23acb87d5e71e773e9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 40717 } } peers { permanent_uuid: "06dc406fe0214c96b029e8f04283e76d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 39077 } }
I20260620 23:08:02.083757 25659 consensus_queue.cc:260] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32777 } } peers { permanent_uuid: "ab1f290bbac84a23acb87d5e71e773e9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 40717 } } peers { permanent_uuid: "06dc406fe0214c96b029e8f04283e76d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 39077 } }
I20260620 23:08:02.088358 25660 ts_tablet_manager.cc:1428] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260620 23:08:02.088824 25352 heartbeater.cc:503] Master 127.12.2.126:37163 was elected leader, sending a full tablet report...
I20260620 23:08:02.089594 25659 ts_tablet_manager.cc:1428] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260620 23:08:02.091809 25661 tablet_bootstrap.cc:492] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: No bootstrap required, opened a new log
I20260620 23:08:02.092144 25661 ts_tablet_manager.cc:1397] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.006s
I20260620 23:08:02.099802 25661 raft_consensus.cc:348] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32777 } } peers { permanent_uuid: "ab1f290bbac84a23acb87d5e71e773e9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 40717 } } peers { permanent_uuid: "06dc406fe0214c96b029e8f04283e76d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 39077 } }
I20260620 23:08:02.100140 25661 raft_consensus.cc:374] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:02.100226 25661 raft_consensus.cc:729] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a27763a9a3b484ea5abd1b53d30cfb2, State: Initialized, Role: FOLLOWER
I20260620 23:08:02.100755 25661 consensus_queue.cc:260] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32777 } } peers { permanent_uuid: "ab1f290bbac84a23acb87d5e71e773e9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 40717 } } peers { permanent_uuid: "06dc406fe0214c96b029e8f04283e76d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 39077 } }
I20260620 23:08:02.101747 25631 heartbeater.cc:503] Master 127.12.2.126:37163 was elected leader, sending a full tablet report...
I20260620 23:08:02.102283 25661 ts_tablet_manager.cc:1428] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 23:08:02.106298 25665 raft_consensus.cc:482] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:08:02.106587 25665 raft_consensus.cc:504] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32777 } } peers { permanent_uuid: "ab1f290bbac84a23acb87d5e71e773e9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 40717 } } peers { permanent_uuid: "06dc406fe0214c96b029e8f04283e76d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 39077 } }
I20260620 23:08:02.108223 25665 leader_election.cc:288] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a27763a9a3b484ea5abd1b53d30cfb2 (127.12.2.67:32777), ab1f290bbac84a23acb87d5e71e773e9 (127.12.2.65:40717)
I20260620 23:08:02.115926 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.109089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:38455 (local address 127.12.2.67:32777)
0620 23:08:02.109245 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:02.109250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:02.110338 (+ 1088us) server_negotiation.cc:407] Connection header received
0620 23:08:02.110486 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:02.110491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:02.110555 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:02.110639 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:02.111228 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.111947 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.113867 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.114981 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.115124 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.115234 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:02.115543 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 23:08:02.115671 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:08:02.116065 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.108603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:32777 (local address 127.12.2.66:38455)
0620 23:08:02.110239 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.110256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:02.110372 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:02.110684 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:02.110689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:02.110719 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:02.111081 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:02.111091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.112083 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.112090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:02.113718 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:02.113731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.115187 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.115194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:02.115358 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.115394 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:08:02.115487 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:08:02.115614 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"mutex_wait_us":50,"thread_start_us":1336,"threads_started":1}
I20260620 23:08:02.116735 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.108832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:40719 (local address 127.12.2.65:40717)
0620 23:08:02.109056 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:02.109064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:02.109670 (+ 606us) server_negotiation.cc:407] Connection header received
0620 23:08:02.109853 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:02.109859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:02.109961 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:02.110063 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:02.110873 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.111952 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.114251 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.115470 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.115626 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.115744 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:02.116333 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 23:08:02.116473 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:08:02.117040 25586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de686616199440b19230adebc909ea38" candidate_uuid: "06dc406fe0214c96b029e8f04283e76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" is_pre_election: true
I20260620 23:08:02.117385 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.108542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:40717 (local address 127.12.2.66:40719)
0620 23:08:02.109563 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.109579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:02.109714 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:02.110252 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:02.110258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:02.110286 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:02.110723 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:02.110735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.112124 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.112129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:02.114089 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:02.114105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.115908 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.115916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:02.116107 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.116142 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 23:08:02.116962 (+ 820us) client_negotiation.cc:240] Negotiation successful
0620 23:08:02.117072 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":23,"spinlock_wait_cycles":17792,"thread_start_us":694,"threads_started":1}
I20260620 23:08:02.117581 25586 raft_consensus.cc:2454] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06dc406fe0214c96b029e8f04283e76d in term 0.
I20260620 23:08:02.118184 25387 leader_election.cc:302] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06dc406fe0214c96b029e8f04283e76d, 9a27763a9a3b484ea5abd1b53d30cfb2; no voters:
I20260620 23:08:02.118574 25665 raft_consensus.cc:2781] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:08:02.118422 25307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de686616199440b19230adebc909ea38" candidate_uuid: "06dc406fe0214c96b029e8f04283e76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1f290bbac84a23acb87d5e71e773e9" is_pre_election: true
I20260620 23:08:02.118737 25665 raft_consensus.cc:482] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:08:02.118857 25665 raft_consensus.cc:3037] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:02.119036 25307 raft_consensus.cc:2454] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06dc406fe0214c96b029e8f04283e76d in term 0.
I20260620 23:08:02.121411 25665 raft_consensus.cc:504] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32777 } } peers { permanent_uuid: "ab1f290bbac84a23acb87d5e71e773e9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 40717 } } peers { permanent_uuid: "06dc406fe0214c96b029e8f04283e76d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 39077 } }
I20260620 23:08:02.122097 25665 leader_election.cc:288] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [CANDIDATE]: Term 1 election: Requested vote from peers 9a27763a9a3b484ea5abd1b53d30cfb2 (127.12.2.67:32777), ab1f290bbac84a23acb87d5e71e773e9 (127.12.2.65:40717)
I20260620 23:08:02.122545 25586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de686616199440b19230adebc909ea38" candidate_uuid: "06dc406fe0214c96b029e8f04283e76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2"
I20260620 23:08:02.122607 25307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de686616199440b19230adebc909ea38" candidate_uuid: "06dc406fe0214c96b029e8f04283e76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1f290bbac84a23acb87d5e71e773e9"
I20260620 23:08:02.122751 25586 raft_consensus.cc:3037] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:02.122787 25307 raft_consensus.cc:3037] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:02.125044 25586 raft_consensus.cc:2454] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06dc406fe0214c96b029e8f04283e76d in term 1.
I20260620 23:08:02.125509 25387 leader_election.cc:302] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06dc406fe0214c96b029e8f04283e76d, 9a27763a9a3b484ea5abd1b53d30cfb2; no voters:
I20260620 23:08:02.126118 25307 raft_consensus.cc:2454] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06dc406fe0214c96b029e8f04283e76d in term 1.
I20260620 23:08:02.126149 25665 raft_consensus.cc:2781] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:08:02.126652 25665 raft_consensus.cc:686] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 1 LEADER]: Becoming Leader. State: Replica: 06dc406fe0214c96b029e8f04283e76d, State: Running, Role: LEADER
I20260620 23:08:02.127236 25665 consensus_queue.cc:237] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32777 } } peers { permanent_uuid: "ab1f290bbac84a23acb87d5e71e773e9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 40717 } } peers { permanent_uuid: "06dc406fe0214c96b029e8f04283e76d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 39077 } }
I20260620 23:08:02.131982 25155 catalog_manager.cc:5495] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d reported cstate change: term changed from 0 to 1, leader changed from <none> to 06dc406fe0214c96b029e8f04283e76d (127.12.2.66). New cstate: current_term: 1 leader_uuid: "06dc406fe0214c96b029e8f04283e76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1f290bbac84a23acb87d5e71e773e9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 40717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06dc406fe0214c96b029e8f04283e76d" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 39077 } health_report { overall_health: HEALTHY } } }
W20260620 23:08:02.143491 25632 tablet.cc:2369] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:08:02.167614 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.162397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.12.2.66:39077)
0620 23:08:02.162582 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:02.162588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:02.162607 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:08:02.162790 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:02.162794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:02.162880 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:02.163034 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:08:02.163379 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.164087 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.165476 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.165874 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.165977 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.166442 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:08:02.167065 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:08:02.167211 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:02.167282 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 23:08:02.167362 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 23:08:02.174448 25498 tablet.cc:2369] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:08:02.175482 25586 raft_consensus.cc:1261] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2 [term 1 FOLLOWER]: Refusing update from remote peer 06dc406fe0214c96b029e8f04283e76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:02.175722 25307 raft_consensus.cc:1261] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9 [term 1 FOLLOWER]: Refusing update from remote peer 06dc406fe0214c96b029e8f04283e76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:02.176537 25665 consensus_queue.cc:1035] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1f290bbac84a23acb87d5e71e773e9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 40717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:02.176832 25670 consensus_queue.cc:1035] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a27763a9a3b484ea5abd1b53d30cfb2" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:02.193670 25673 mvcc.cc:205] Tried to move back new op lower bound from 7299059229380583424 to 7299059229197176832. Current Snapshot: MvccSnapshot[applied={T|T < 7299059229380583424}]
I20260620 23:08:02.217128 25155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:08:02.217365 25155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:08:02.220580 25155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fdf57ebc400408b977fbce4aec70cf3: Sending DeleteTablet for 3 replicas of tablet de686616199440b19230adebc909ea38
I20260620 23:08:02.221657 25566 tablet_service.cc:1514] Processing DeleteTablet for tablet de686616199440b19230adebc909ea38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:02 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:56888
I20260620 23:08:02.221972 25287 tablet_service.cc:1514] Processing DeleteTablet for tablet de686616199440b19230adebc909ea38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:02 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:53226
I20260620 23:08:02.222609 25431 tablet_service.cc:1514] Processing DeleteTablet for tablet de686616199440b19230adebc909ea38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:02 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:58462
I20260620 23:08:02.233489 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 25192
I20260620 23:08:02.256860 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.234827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.12.2.65:40717)
0620 23:08:02.235044 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:02.235052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:02.236137 (+ 1085us) server_negotiation.cc:407] Connection header received
0620 23:08:02.236213 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:02.236220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:02.236325 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:02.236432 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:02.238997 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.240118 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.241437 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.242128 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.242252 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.246181 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:02.246198 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:02.246219 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:02.246268 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:02.250074 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:02.252403 (+ 2329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:02.252412 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:02.252416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:02.252613 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:02.253383 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:02.253388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:02.253391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:02.253833 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:02.254127 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:02.256237 (+ 2110us) server_negotiation.cc:299] Negotiation successful
0620 23:08:02.256456 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:08:02.257949 25685 tablet_replica.cc:330] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: stopping tablet replica
I20260620 23:08:02.258404 25685 raft_consensus.cc:2229] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:08:02.259052 25685 raft_consensus.cc:2258] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:02.259104 25347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:02.261333 25685 ts_tablet_manager.cc:1905] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:02.271131 25683 tablet_replica.cc:330] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: stopping tablet replica
I20260620 23:08:02.271551 25683 raft_consensus.cc:2229] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:08:02.272038 25683 raft_consensus.cc:2258] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:02.282426 25683 ts_tablet_manager.cc:1905] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:02.289486 25685 ts_tablet_manager.cc:1918] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:02.289686 25685 log.cc:1196] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/de686616199440b19230adebc909ea38
I20260620 23:08:02.290179 25685 ts_tablet_manager.cc:1939] T de686616199440b19230adebc909ea38 P 06dc406fe0214c96b029e8f04283e76d: Deleting consensus metadata
I20260620 23:08:02.291360 25683 ts_tablet_manager.cc:1918] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:02.291517 25683 log.cc:1196] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/de686616199440b19230adebc909ea38
I20260620 23:08:02.292003 25683 ts_tablet_manager.cc:1939] T de686616199440b19230adebc909ea38 P 9a27763a9a3b484ea5abd1b53d30cfb2: Deleting consensus metadata
I20260620 23:08:02.292522 25141 catalog_manager.cc:4841] TS 06dc406fe0214c96b029e8f04283e76d (127.12.2.66:39077): tablet de686616199440b19230adebc909ea38 (table test-table [id=ded3744606b3450aa68f76189a1b6a81]) successfully deleted
I20260620 23:08:02.293774 25142 catalog_manager.cc:4841] TS 9a27763a9a3b484ea5abd1b53d30cfb2 (127.12.2.67:32777): tablet de686616199440b19230adebc909ea38 (table test-table [id=ded3744606b3450aa68f76189a1b6a81]) successfully deleted
I20260620 23:08:02.460289 25684 tablet_replica.cc:330] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9: stopping tablet replica
I20260620 23:08:02.460795 25684 raft_consensus.cc:2229] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:08:02.461335 25684 raft_consensus.cc:2258] T de686616199440b19230adebc909ea38 P ab1f290bbac84a23acb87d5e71e773e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:02.462878 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 25192
I20260620 23:08:02.484594 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 25356
W20260620 23:08:02.484829 25139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.2.65:40717 (error 108)
I20260620 23:08:02.487743 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.487245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:40717 (local address 127.0.0.1:53242)
0620 23:08:02.487450 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.487578 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260620 23:08:02.488130 25139 catalog_manager.cc:4584] TS ab1f290bbac84a23acb87d5e71e773e9 (127.12.2.65:40717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de686616199440b19230adebc909ea38: Network error: Client connection negotiation failed: client connection to 127.12.2.65:40717: connect: Connection refused (error 111)
I20260620 23:08:02.502465 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.485690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.12.2.66:39077)
0620 23:08:02.485852 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:02.485858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:02.485875 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:08:02.485966 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:02.485972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:02.486041 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:02.486140 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:02.487960 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.489177 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.491142 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.491827 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.491951 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.495219 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:02.495231 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:02.495247 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:02.495294 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:02.499285 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:02.500173 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:02.500183 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:02.500186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:02.500374 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:02.500792 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:02.500797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:02.500801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:02.501235 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:02.501526 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:02.501846 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 23:08:02.502089 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:08:02.503342 25492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:02.541538 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.541086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:40717 (local address 127.0.0.1:53258)
0620 23:08:02.541302 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.541399 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 23:08:02.542471 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.542155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:40717 (local address 127.0.0.1:53264)
0620 23:08:02.542292 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.542357 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 23:08:02.603053 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.602469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:40717 (local address 127.0.0.1:53274)
0620 23:08:02.602699 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.602814 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:08:02.604223 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.603837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:40717 (local address 127.0.0.1:53280)
0620 23:08:02.603991 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.604077 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 23:08:02.614466 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 25356
I20260620 23:08:02.628981 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 25501
I20260620 23:08:02.641817 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.629732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.12.2.67:32777)
0620 23:08:02.630165 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:02.630169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:02.630184 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:08:02.630233 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:02.630236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:02.630301 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:02.630388 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:02.631709 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.632402 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.633680 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:02.634069 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:02.634174 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:02.636631 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:02.636643 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:02.636660 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:02.636692 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:02.639241 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:02.639893 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:02.639901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:02.639903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:02.640026 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:02.640371 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:02.640375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:02.640377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:02.640713 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:02.640982 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:02.641271 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 23:08:02.641472 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":212,"threads_started":1}
I20260620 23:08:02.642467 25626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:02.684814 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.684270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:40717 (local address 127.0.0.1:53282)
0620 23:08:02.684502 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.684637 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 23:08:02.685829 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:02.685441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:40717 (local address 127.0.0.1:53284)
0620 23:08:02.685660 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:02.685726 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 23:08:02.731268 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 25501
I20260620 23:08:02.744045 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 25122
I20260620 23:08:02.749078 25185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:02.857362 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 25122
I20260620 23:08:02.874358 12297 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:08:03.209438 12297 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:08:03.209693 12297 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:08:03.545018 12297 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:08:03.545 UTC [24175] LOG: received smart shutdown request
2026-06-20 23:08:03.548 UTC [24175] LOG: background worker "logical replication launcher" (PID 24184) exited with exit code 1
2026-06-20 23:08:03.548 UTC [24179] LOG: shutting down
2026-06-20 23:08:03.685 UTC [24175] LOG: database system is shut down
2026-06-20T23:08:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49967 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:08:03 dist-test-slave-7g8g krb5kdc[25700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:08:03 dist-test-slave-7g8g krb5kdc[25700](info): set up 2 sockets
Jun 20 23:08:03 dist-test-slave-7g8g krb5kdc[25700](info): commencing operation
krb5kdc: starting...
W20260620 23:08:04.794888 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.008s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:08:04 dist-test-slave-7g8g krb5kdc[25700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:08:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:08:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:08:05.006776 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:38115
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:38115,127.12.2.125:34193,127.12.2.124:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:05.321233 25713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:05.321522 25713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:05.321659 25713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:05.328763 25713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:08:05.328877 25713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:05.328956 25713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:05.329028 25713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:08:05.329087 25713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:08:05.338651 25713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:38115,127.12.2.125:34193,127.12.2.124:39091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:38115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:05.339915 25713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:05.341202 25713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:05.351975 25718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:05.352046 25719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:05.352118 25721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:05.352973 25713 server_base.cc:1034] running on GCE node
I20260620 23:08:05.353703 25713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:05.355007 25713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:05.356240 25713 hybrid_clock.cc:648] HybridClock initialized: now 1781996885356184 us; error 44 us; skew 500 ppm
I20260620 23:08:05.356631 25713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:05 dist-test-slave-7g8g krb5kdc[25700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996885, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:05.359639 25713 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:08:05.361099 25713 webserver.cc:460] Webserver started at http://127.12.2.126:41933/ using document root <none> and password file <none>
I20260620 23:08:05.361784 25713 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:05.361898 25713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:05.362177 25713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:05.365075 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "18c1b242306b477581bfa02677256db0"
format_stamp: "Formatted at 2026-06-20 23:08:05 on dist-test-slave-7g8g"
server_key: "c52fdde51ee4f419727a7235fa53bac2"
server_key_iv: "a1bd1a9ad72e4e5ca4c65a300e33e0fc"
server_key_version: "encryptionkey@0"
I20260620 23:08:05.365878 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "18c1b242306b477581bfa02677256db0"
format_stamp: "Formatted at 2026-06-20 23:08:05 on dist-test-slave-7g8g"
server_key: "c52fdde51ee4f419727a7235fa53bac2"
server_key_iv: "a1bd1a9ad72e4e5ca4c65a300e33e0fc"
server_key_version: "encryptionkey@0"
I20260620 23:08:05.370782 25713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:08:05.374009 25728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:05.375276 25713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:08:05.375420 25713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "18c1b242306b477581bfa02677256db0"
format_stamp: "Formatted at 2026-06-20 23:08:05 on dist-test-slave-7g8g"
server_key: "c52fdde51ee4f419727a7235fa53bac2"
server_key_iv: "a1bd1a9ad72e4e5ca4c65a300e33e0fc"
server_key_version: "encryptionkey@0"
I20260620 23:08:05.375597 25713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:05.391177 25713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:05.394860 25713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:05.395139 25713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:05.411815 25713 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:38115
I20260620 23:08:05.411845 25779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:38115 every 8 connection(s)
I20260620 23:08:05.413115 25713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:08:05.416662 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 25713
I20260620 23:08:05.416747 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:05.416915 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:08:05.417703 12297 external_mini_cluster.cc:1428] Setting key ef05f7cf34cede335850581fd07990e8
I20260620 23:08:05.425643 25780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.2.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.125@KRBTEST.COM" created.
Jun 20 23:08:05 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996885, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:05.449781 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.430700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.12.2.126:38115)
0620 23:08:05.431406 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:05.431415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:05.431472 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:08:05.432566 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:05.432572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:05.432814 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:05.432990 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:05.434257 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.435592 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.436949 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.437511 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.437691 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:05.442342 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:05.442354 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:05.442373 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:05.442413 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:05.445717 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:05.446424 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:05.446433 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:05.446436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:05.446669 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:05.447069 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:05.447073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:05.447076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:05.447738 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:05.447935 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:05.448471 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 23:08:05.448720 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":505,"thread_start_us":333,"threads_started":1}
I20260620 23:08:05.450328 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.429271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:38115 (local address 127.0.0.1:34168)
0620 23:08:05.430118 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.430157 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:05.432195 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:05.433032 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:05.433039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:05.433721 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:05.434065 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:05.434082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.435735 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.435739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:05.436820 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:05.436832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.437743 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.437750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:05.438011 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:05.438652 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:05.438675 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:05.442041 (+ 3366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:05.445866 (+ 3825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:05.445877 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:05.445887 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:05.446292 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:05.446788 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:05.446791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:05.446794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:05.446981 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:05.447860 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:05.447866 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:05.448062 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 23:08:05.448288 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 23:08:05.448886 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":159,"thread_start_us":367,"threads_started":1}
I20260620 23:08:05.453236 25780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:08:05.454301 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.453870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54228)
0620 23:08:05.454046 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.454171 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 23:08:05.454574 25731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
I20260620 23:08:05.456136 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.455784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54236)
0620 23:08:05.455910 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.455987 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 23:08:05.456316 25780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:34193: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab.
Entry for principal kudu/127.12.2.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab.
I20260620 23:08:05.468175 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.125:34193
--webserver_interface=127.12.2.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab
--principal=kudu/127.12.2.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:38115,127.12.2.125:34193,127.12.2.124:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:08:05.505702 25780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } attempt: 1
I20260620 23:08:05.506966 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.506404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54242)
0620 23:08:05.506644 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.506781 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 23:08:05.508096 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.507658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54252)
0620 23:08:05.507815 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.507908 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:08:05.508313 25780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:34193: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
I20260620 23:08:05.576576 25780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } attempt: 2
I20260620 23:08:05.577828 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.577319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54262)
0620 23:08:05.577533 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.577653 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 23:08:05.578858 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.578457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54278)
0620 23:08:05.578597 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.578703 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 23:08:05.579079 25780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:34193: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
I20260620 23:08:05.670356 25780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } attempt: 3
I20260620 23:08:05.671698 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.671148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54282)
0620 23:08:05.671368 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.671502 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 23:08:05.672837 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.672422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54288)
0620 23:08:05.672576 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.672671 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:08:05.673045 25780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:34193: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
W20260620 23:08:05.775121 25787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:05.775389 25787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:05.775490 25787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:05.782586 25787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:08:05.782689 25787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:05.782742 25787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:05.782795 25787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:08:05.782840 25787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:08:05.792357 25787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:38115,127.12.2.125:34193,127.12.2.124:39091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab
--principal=kudu/127.12.2.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.125:34193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.2.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:05.793653 25787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:05.794880 25787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:05.806000 25795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:05.805979 25793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:05.806330 25792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:05.806730 25787 server_base.cc:1034] running on GCE node
I20260620 23:08:05.807475 25787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:05.808751 25787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:05.809937 25787 hybrid_clock.cc:648] HybridClock initialized: now 1781996885809885 us; error 41 us; skew 500 ppm
I20260620 23:08:05.810237 25787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:08:05.816273 25780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } attempt: 4
Jun 20 23:08:05 dist-test-slave-7g8g krb5kdc[25700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996885, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:05.816879 25787 init.cc:377] Logged in from keytab as kudu/127.12.2.125@KRBTEST.COM (short username kudu)
I20260620 23:08:05.817411 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.816803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54296)
0620 23:08:05.817109 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.817221 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260620 23:08:05.818432 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.818038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54298)
0620 23:08:05.818179 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.818283 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:08:05.818663 25787 webserver.cc:460] Webserver started at http://127.12.2.125:35729/ using document root <none> and password file <none>
W20260620 23:08:05.818672 25780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:34193: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
I20260620 23:08:05.819398 25787 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:05.819510 25787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:05.819794 25787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:05.822697 25787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/data/instance:
uuid: "e477ee440bc94f71898ecf02474b070c"
format_stamp: "Formatted at 2026-06-20 23:08:05 on dist-test-slave-7g8g"
server_key: "f47e56d16253cd0e59ac37011871161b"
server_key_iv: "b74b943cce329c997662702e047cf957"
server_key_version: "encryptionkey@0"
I20260620 23:08:05.823483 25787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/wal/instance:
uuid: "e477ee440bc94f71898ecf02474b070c"
format_stamp: "Formatted at 2026-06-20 23:08:05 on dist-test-slave-7g8g"
server_key: "f47e56d16253cd0e59ac37011871161b"
server_key_iv: "b74b943cce329c997662702e047cf957"
server_key_version: "encryptionkey@0"
I20260620 23:08:05.828299 25787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:08:05.831542 25802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:05.832821 25787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:08:05.832964 25787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/wal
uuid: "e477ee440bc94f71898ecf02474b070c"
format_stamp: "Formatted at 2026-06-20 23:08:05 on dist-test-slave-7g8g"
server_key: "f47e56d16253cd0e59ac37011871161b"
server_key_iv: "b74b943cce329c997662702e047cf957"
server_key_version: "encryptionkey@0"
I20260620 23:08:05.833137 25787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:05.856143 25787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:05.860327 25787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:05.860579 25787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:05.877482 25787 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.125:34193
I20260620 23:08:05.877516 25853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.125:34193 every 8 connection(s)
I20260620 23:08:05.878755 25787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
I20260620 23:08:05.882396 25854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:05.886955 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 25787
I20260620 23:08:05.887127 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-1/wal/instance
I20260620 23:08:05.887519 12297 external_mini_cluster.cc:1428] Setting key de547cfb4879e72473861d2b325b3c31
I20260620 23:08:05.888584 25854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:08:05 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996885, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.2.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.124@KRBTEST.COM" created.
I20260620 23:08:05.908242 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.891064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.12.2.126:38115)
0620 23:08:05.891227 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:05.891231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:05.892638 (+ 1407us) server_negotiation.cc:407] Connection header received
0620 23:08:05.894468 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:05.894474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:05.894555 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:05.894643 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:05.896140 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.896819 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.898377 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.898779 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.898887 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:05.902307 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:05.902314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:05.902319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:05.902363 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:05.905486 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:05.906192 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:05.906196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:05.906199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:05.906317 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:05.906653 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:05.906657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:05.906659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:05.906973 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:05.907099 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:05.907772 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 23:08:05.907972 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:08:05.909209 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.891682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:38115 (local address 127.0.0.1:34172)
0620 23:08:05.892449 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.892514 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:05.894197 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:05.894873 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:05.894881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:05.895409 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:05.895949 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:05.895970 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.896966 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.896970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:05.898235 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:05.898247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.899010 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.899017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:05.899228 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:05.899820 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:05.899847 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:05.902063 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:05.905618 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:05.905625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:05.905640 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:05.906056 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:05.906401 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:05.906404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:05.906406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:05.906556 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:05.907075 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:05.907080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:05.907395 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 23:08:05.907610 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 23:08:05.908137 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":325,"threads_started":1}
I20260620 23:08:05.910877 25854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:08:05 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996885, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
I20260620 23:08:05.923882 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.911626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54302)
0620 23:08:05.911784 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.911801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:05.911885 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:05.912778 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:05.912782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:05.913151 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:05.913270 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:05.913278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.914531 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.914536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:05.915387 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:05.915398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.916137 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.916143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:05.916345 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:05.916851 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:05.916867 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:05.918860 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:05.921627 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:05.921632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:05.921637 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:05.921949 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:05.922271 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:05.922275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:05.922279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:05.922416 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:05.922988 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:05.922993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:05.923125 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 23:08:05.923239 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:08:05.923487 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:08:05.923964 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.911835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.12.2.125:34193)
0620 23:08:05.912407 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:05.912413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:05.912437 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:08:05.912474 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:05.912479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:05.912649 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:05.912769 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:05.913408 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.914411 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.915507 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:05.916013 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:05.916125 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:05.919045 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:05.919055 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:05.919067 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:05.919096 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:05.921486 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:05.922059 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:05.922066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:05.922068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:05.922177 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:05.922506 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:05.922511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:05.922514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:05.922853 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:05.923073 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:05.923394 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:08:05.923589 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":441,"thread_start_us":318,"threads_started":1}
I20260620 23:08:05.926131 25854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:08:05.927353 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.926880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45840)
0620 23:08:05.927100 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.927217 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 23:08:05.927651 25806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
I20260620 23:08:05.929229 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.928800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45856)
0620 23:08:05.928946 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.929048 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 23:08:05.929437 25854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:39091: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab.
Entry for principal kudu/127.12.2.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab.
I20260620 23:08:05.932166 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.124:39091
--webserver_interface=127.12.2.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab
--principal=kudu/127.12.2.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:38115,127.12.2.125:34193,127.12.2.124:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:08:05.978735 25854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } attempt: 1
I20260620 23:08:05.980011 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.979512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45864)
0620 23:08:05.979738 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.979846 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 23:08:05.981104 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:05.980748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45872)
0620 23:08:05.980908 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:05.980984 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:08:05.981314 25854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:39091: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
I20260620 23:08:06.049535 25854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } attempt: 2
I20260620 23:08:06.050743 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.050244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45876)
0620 23:08:06.050439 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.050534 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 23:08:06.051790 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.051468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45884)
0620 23:08:06.051601 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.051673 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 23:08:06.052011 25854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:39091: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
I20260620 23:08:06.117925 25780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } attempt: 5
Jun 20 23:08:06 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996885, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
I20260620 23:08:06.130336 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.118654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54308)
0620 23:08:06.118858 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.118879 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:06.119039 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:06.119415 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:06.119420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:06.119805 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:06.119974 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.119983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.120941 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.120945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.121792 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.121804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.122537 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.122543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.122709 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.123277 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:06.123295 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:06.125225 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:06.128140 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.128146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.128150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.128486 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.128811 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.128816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.128819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.128952 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.129416 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:06.129421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:06.129561 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 23:08:06.129700 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:08:06.129971 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 23:08:06.130338 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.118733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.12.2.125:34193)
0620 23:08:06.118897 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.118902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.118955 (+ 53us) server_negotiation.cc:407] Connection header received
0620 23:08:06.119202 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.119207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.119271 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.119354 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.120133 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.120817 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.121926 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.122389 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.122495 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.125406 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.125411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.125414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.125445 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.128022 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.128597 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.128601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.128603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.128714 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.129045 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.129050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.129053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.129313 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.129437 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.129818 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.130026 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:08:06.131647 25780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:08:06.132709 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.132347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45894)
0620 23:08:06.132495 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.132567 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:08:06.133594 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.133299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45902)
0620 23:08:06.133427 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.133497 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 23:08:06.133783 25780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:39091: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
I20260620 23:08:06.143234 25854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } attempt: 3
I20260620 23:08:06.144253 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.143799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45912)
0620 23:08:06.144005 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.144100 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 23:08:06.145256 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.144834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45926)
0620 23:08:06.145058 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.145137 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260620 23:08:06.145462 25854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:39091: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
I20260620 23:08:06.184981 25780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } attempt: 1
I20260620 23:08:06.186120 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.185662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45936)
0620 23:08:06.185842 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.185947 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 23:08:06.187242 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.186824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45940)
0620 23:08:06.187036 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.187114 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 23:08:06.187464 25780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:39091: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
W20260620 23:08:06.239144 25861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:06.239427 25861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:06.239569 25861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:06.246553 25861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:08:06.246660 25861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:06.246738 25861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:06.246811 25861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:08:06.246870 25861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:08:06.255685 25780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } attempt: 2
I20260620 23:08:06.256438 25861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:38115,127.12.2.125:34193,127.12.2.124:39091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab
--principal=kudu/127.12.2.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.124:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.2.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:06.256796 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.256300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45948)
0620 23:08:06.256517 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.256615 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 23:08:06.257779 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.257484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45956)
0620 23:08:06.257609 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.257680 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 23:08:06.257951 25861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260620 23:08:06.257974 25780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:39091: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
I20260620 23:08:06.259235 25861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:06.269847 25866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:06.269865 25869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:06.269865 25867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:06.271047 25861 server_base.cc:1034] running on GCE node
I20260620 23:08:06.271751 25861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:06.273072 25861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:06.274288 25861 hybrid_clock.cc:648] HybridClock initialized: now 1781996886274244 us; error 47 us; skew 500 ppm
I20260620 23:08:06.274654 25861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:06 dist-test-slave-7g8g krb5kdc[25700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996886, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:06.277665 25861 init.cc:377] Logged in from keytab as kudu/127.12.2.124@KRBTEST.COM (short username kudu)
I20260620 23:08:06.279233 25861 webserver.cc:460] Webserver started at http://127.12.2.124:36119/ using document root <none> and password file <none>
I20260620 23:08:06.279943 25861 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:06.280054 25861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:06.280366 25861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:06.283368 25861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/data/instance:
uuid: "cb56139364f44f35bfcaea5daa4e2924"
format_stamp: "Formatted at 2026-06-20 23:08:06 on dist-test-slave-7g8g"
server_key: "a8b7e2486f97f792639860493e8c28a2"
server_key_iv: "2e9a514d6cf24ad00f0e056bbe4329c1"
server_key_version: "encryptionkey@0"
I20260620 23:08:06.284148 25861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/wal/instance:
uuid: "cb56139364f44f35bfcaea5daa4e2924"
format_stamp: "Formatted at 2026-06-20 23:08:06 on dist-test-slave-7g8g"
server_key: "a8b7e2486f97f792639860493e8c28a2"
server_key_iv: "2e9a514d6cf24ad00f0e056bbe4329c1"
server_key_version: "encryptionkey@0"
I20260620 23:08:06.288650 25854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } attempt: 4
I20260620 23:08:06.289191 25861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:08:06.289599 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.289152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45968)
0620 23:08:06.289357 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.289439 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:08:06.290488 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.290157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:45984)
0620 23:08:06.290290 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.290364 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 23:08:06.290668 25854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:39091: Network error: Client connection negotiation failed: client connection to 127.12.2.124:39091: connect: Connection refused (error 111)
I20260620 23:08:06.292819 25876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:06.294216 25861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:08:06.294378 25861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/wal
uuid: "cb56139364f44f35bfcaea5daa4e2924"
format_stamp: "Formatted at 2026-06-20 23:08:06 on dist-test-slave-7g8g"
server_key: "a8b7e2486f97f792639860493e8c28a2"
server_key_iv: "2e9a514d6cf24ad00f0e056bbe4329c1"
server_key_version: "encryptionkey@0"
I20260620 23:08:06.294565 25861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:06.324229 25861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:06.328423 25861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:06.328696 25861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:06.345584 25861 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.124:39091
I20260620 23:08:06.345664 25927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.124:39091 every 8 connection(s)
I20260620 23:08:06.346874 25861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
I20260620 23:08:06.350581 25928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:06.351743 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 25861
I20260620 23:08:06.351945 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996696601630-12297-0/minicluster-data/master-2/wal/instance
I20260620 23:08:06.352589 12297 external_mini_cluster.cc:1428] Setting key 829dc86245bdddb849b24a6314a60288
I20260620 23:08:06.359699 25928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:08:06.364158 25780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } attempt: 3
Jun 20 23:08:06 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
Jun 20 23:08:06 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996886, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
Jun 20 23:08:06 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996885, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
I20260620 23:08:06.376575 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.354633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.12.2.124:39091)
0620 23:08:06.355545 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.355584 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.355631 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:08:06.357214 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.357248 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.357744 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.358140 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.359558 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.360867 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.362453 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.363513 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.363690 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.368123 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.368137 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.368156 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.368206 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.371445 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.372403 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.372413 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.372423 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.372625 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.373180 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.373185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.373188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.373890 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.374144 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.374572 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.374998 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":470,"threads_started":1}
I20260620 23:08:06.380334 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.362103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:38115 (local address 127.0.0.1:34182)
0620 23:08:06.362872 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.362919 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:06.363120 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:06.363458 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:06.363465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:06.364170 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:06.364485 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.364495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.365921 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.365926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.367476 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.367486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.368372 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.368379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.368576 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.370431 (+ 1855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:06.370452 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:06.372860 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:06.377449 (+ 4589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.377458 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.377464 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.378028 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.378395 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.378400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.378404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.378608 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.379238 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:06.379246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:06.379460 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 23:08:06.379646 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 23:08:06.379970 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":24,"thread_start_us":372,"threads_started":1}
I20260620 23:08:06.380566 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.364711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:46000)
0620 23:08:06.364960 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.364976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:06.365071 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:06.367134 (+ 2063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:06.367139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:06.367597 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:06.367850 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.367865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.369262 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.369268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.370157 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.370169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.371103 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.371109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.371289 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.371831 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:06.371853 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:06.374172 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:06.377916 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.377920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.377924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.378383 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.378804 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.378809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.378812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.378977 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.379600 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:06.379605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:06.379772 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 23:08:06.379899 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:08:06.380141 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":6}
I20260620 23:08:06.381522 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.362072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.12.2.126:38115)
0620 23:08:06.362242 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.362248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.363062 (+ 814us) server_negotiation.cc:407] Connection header received
0620 23:08:06.363229 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.363233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.363296 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.363373 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.364618 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.365748 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.367604 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.368188 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.368316 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.373089 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.373095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.373100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.373139 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.377326 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.378148 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.378152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.378154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.378289 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.378718 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.378721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.378723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.379133 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.379272 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.380956 (+ 1684us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.381164 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":50}
I20260620 23:08:06.381374 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.364749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.12.2.124:39091)
0620 23:08:06.366747 (+ 1998us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.366752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.366773 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:08:06.366814 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.366819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.366879 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.367069 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.368037 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.369146 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.370288 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.370907 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.371077 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.374362 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.374368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.374372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.374403 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.377761 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.378540 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.378544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.378547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.378694 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.379111 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.379115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.379119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.379467 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.380785 (+ 1318us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.380904 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.381109 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1873,"thread_start_us":1715,"threads_started":1}
I20260620 23:08:06.383240 25928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:08:06.387542 25780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0: Bootstrap starting.
Jun 20 23:08:06 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996886, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
Jun 20 23:08:06 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:06.390923 25780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:06.392251 25780 log.cc:824] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:06.395682 25780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0: No bootstrap required, opened a new log
I20260620 23:08:06.397085 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.383904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.12.2.125:34193)
0620 23:08:06.384077 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.384083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.384131 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:08:06.384286 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.384292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.384359 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.384436 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.385073 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.385718 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.387066 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.387483 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.387591 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.391618 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.391625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.391629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.391671 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.394460 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.395123 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.395128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.395132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.395266 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.395710 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.395715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.395718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.396065 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.396169 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.396651 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.396810 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:08:06.398017 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.383829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:54324)
0620 23:08:06.384068 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.384085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:06.384178 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:06.384466 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:06.384470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:06.384806 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:06.384932 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.384939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.385840 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.385845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.386906 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.386921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.387618 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.387624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.387833 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.388354 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:06.388376 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:06.391420 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:06.394591 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.394596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.394601 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.395001 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.395392 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.395397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.395400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.395587 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.396232 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:06.396235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:06.396375 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 23:08:06.397339 (+ 964us) client_negotiation.cc:240] Negotiation successful
0620 23:08:06.397709 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260620 23:08:06.398387 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.380005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.12.2.126:38115)
0620 23:08:06.380641 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.380647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.380669 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:08:06.380713 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.380717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.380781 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.380884 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.382435 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.383704 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.385472 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.386105 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.386236 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.391776 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.391782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.391787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.391822 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.395235 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.396071 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.396077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.396080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.396240 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.396654 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.396659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.396662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.396985 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.397519 (+ 534us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.397648 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.397862 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":520,"threads_started":1}
I20260620 23:08:06.399104 25928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:08:06.405457 25780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } }
I20260620 23:08:06.405869 25780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:06.405993 25780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c1b242306b477581bfa02677256db0, State: Initialized, Role: FOLLOWER
I20260620 23:08:06.406780 25780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } }
Jun 20 23:08:06 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996886, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
I20260620 23:08:06.407852 25936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } } }
I20260620 23:08:06.408311 25936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:08:06.409195 25780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 23:08:06 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
I20260620 23:08:06.414343 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.400065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.12.2.124:39091)
0620 23:08:06.400254 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.400260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.400277 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:08:06.400321 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.400326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.400389 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.400476 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.401349 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.402364 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.403950 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.404606 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.404729 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.408680 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.408686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.408691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.408726 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.411622 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.412349 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.412354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.412358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.412488 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.412889 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.412894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.412897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.413227 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.413338 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.413819 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.414022 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:08:06.415210 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.399882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:46004)
0620 23:08:06.400026 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.400043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:06.400143 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:06.400602 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:06.400607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:06.401024 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:06.401202 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.401212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.402495 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.402500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.403807 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.403821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.404771 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.404777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.404952 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.405982 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:06.406005 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:06.408476 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:06.411765 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.411769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.411773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.412234 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.412629 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.412633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.412636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.412780 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.413415 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:06.413420 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:06.413568 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 23:08:06.414595 (+ 1027us) client_negotiation.cc:240] Negotiation successful
0620 23:08:06.414864 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:08:06.418125 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.400735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.12.2.125:34193)
0620 23:08:06.400865 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.400870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.400977 (+ 107us) server_negotiation.cc:407] Connection header received
0620 23:08:06.401129 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.401134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.401192 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.402478 (+ 1286us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.402581 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.403310 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.404836 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.405289 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.405400 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.410762 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.410768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.410773 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.410809 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.414075 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.415163 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.415169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.415172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.415329 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.416452 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.416456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.416459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.416793 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.417022 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.417588 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.417807 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:08:06.421015 25928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924: Bootstrap starting.
I20260620 23:08:06.423898 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:08:06.425035 25928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:06.426249 25928 log.cc:824] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:06.435815 25928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924: No bootstrap required, opened a new log
I20260620 23:08:06.435503 25713 master_runner.cc:186] Error getting master registration for 127.12.2.125:34193: OK, instance_id { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" instance_seqno: 1781996885872735 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:08:06.446444 25713 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:08:06.447678 25928 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } }
I20260620 23:08:06.448232 25928 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:06.448414 25928 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb56139364f44f35bfcaea5daa4e2924, State: Initialized, Role: FOLLOWER
I20260620 23:08:06.449251 25928 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } }
I20260620 23:08:06.450095 25955 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } } }
I20260620 23:08:06.450517 25955 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:08:06.451417 25928 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:08:06.453478 25949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:08:06.453660 25949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:08:06.475585 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.448968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.12.2.125:34193)
0620 23:08:06.449152 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.449158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.457293 (+ 8135us) server_negotiation.cc:407] Connection header received
0620 23:08:06.457593 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.457601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.457692 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.457768 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.460260 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.461454 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.463697 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.464384 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.464480 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.468347 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.468355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.468361 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.468407 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.472055 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.473073 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.473079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.473083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.473249 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.473698 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.473704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.473708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.474110 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.474355 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.474912 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.475207 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:08:06.484228 25968 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:08:06.490453 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.455601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.12.2.126:38115)
0620 23:08:06.455776 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.455782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.463907 (+ 8125us) server_negotiation.cc:407] Connection header received
0620 23:08:06.464932 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.464938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.465041 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.465135 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.470274 (+ 5139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.471425 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.476632 (+ 5207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.477341 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.477451 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.483448 (+ 5997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.483456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.483462 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.483507 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.487023 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.487953 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.487959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.487962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.488263 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.488684 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.488690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.488693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.489042 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.489251 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.489877 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.490099 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:08:06.490453 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.454994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.12.2.124:39091)
0620 23:08:06.455199 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.455205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.466960 (+ 11755us) server_negotiation.cc:407] Connection header received
0620 23:08:06.468501 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.468507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.468612 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.468684 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.473466 (+ 4782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.474552 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.478608 (+ 4056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.479373 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.479470 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.483381 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.483387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.483392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.483430 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.486950 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.487949 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.487955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.487959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.488124 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.488684 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.488690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.488693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.489022 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.489270 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.489878 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.490095 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:08:06.491237 25861 master_runner.cc:186] Error getting master registration for 127.12.2.125:34193: OK, instance_id { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" instance_seqno: 1781996885872735 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:08:06.498901 25861 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:08:06.527523 25955 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:08:06.527842 25955 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } }
I20260620 23:08:06.529559 25955 leader_election.cc:288] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c1b242306b477581bfa02677256db0 (127.12.2.126:38115), e477ee440bc94f71898ecf02474b070c (127.12.2.125:34193)
I20260620 23:08:06.530369 25755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb56139364f44f35bfcaea5daa4e2924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c1b242306b477581bfa02677256db0" is_pre_election: true
I20260620 23:08:06.530882 25755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb56139364f44f35bfcaea5daa4e2924 in term 0.
I20260620 23:08:06.531438 25878 leader_election.cc:302] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c1b242306b477581bfa02677256db0, cb56139364f44f35bfcaea5daa4e2924; no voters:
I20260620 23:08:06.531813 25955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:08:06.531961 25955 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:08:06.532051 25955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:06.534124 25955 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } }
I20260620 23:08:06.534652 25955 leader_election.cc:288] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [CANDIDATE]: Term 1 election: Requested vote from peers 18c1b242306b477581bfa02677256db0 (127.12.2.126:38115), e477ee440bc94f71898ecf02474b070c (127.12.2.125:34193)
I20260620 23:08:06.535210 25755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb56139364f44f35bfcaea5daa4e2924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c1b242306b477581bfa02677256db0"
I20260620 23:08:06.535444 25755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:06.538784 25755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb56139364f44f35bfcaea5daa4e2924 in term 1.
I20260620 23:08:06.539324 25878 leader_election.cc:302] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c1b242306b477581bfa02677256db0, cb56139364f44f35bfcaea5daa4e2924; no voters:
I20260620 23:08:06.539971 25955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:08:06.540470 25955 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [term 1 LEADER]: Becoming Leader. State: Replica: cb56139364f44f35bfcaea5daa4e2924, State: Running, Role: LEADER
I20260620 23:08:06.541018 25955 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } }
I20260620 23:08:06.544049 25969 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb56139364f44f35bfcaea5daa4e2924. Latest consensus state: current_term: 1 leader_uuid: "cb56139364f44f35bfcaea5daa4e2924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } } }
I20260620 23:08:06.544487 25969 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:06.545594 25971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:08:06.558370 25971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:08:06.567626 25755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [term 1 FOLLOWER]: Refusing update from remote peer cb56139364f44f35bfcaea5daa4e2924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:06.568536 25969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:06.580430 25936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb56139364f44f35bfcaea5daa4e2924. Latest consensus state: current_term: 1 leader_uuid: "cb56139364f44f35bfcaea5daa4e2924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } } }
I20260620 23:08:06.580744 25936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:08:06.580736 25882 tablet.cc:2369] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:08:06.584851 25973 mvcc.cc:205] Tried to move back new op lower bound from 7299059247369302016 to 7299059247276793856. Current Snapshot: MvccSnapshot[applied={T|T < 7299059247369302016}]
I20260620 23:08:06.585749 25955 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb56139364f44f35bfcaea5daa4e2924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } } }
I20260620 23:08:06.586061 25955 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:06.588907 25971 catalog_manager.cc:1347] Generated new cluster ID: 1922bdf9f54a42f09981f2c24a6512cf
I20260620 23:08:06.588997 25971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:08:06.589861 25854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } attempt: 5
I20260620 23:08:06.599570 25936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb56139364f44f35bfcaea5daa4e2924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } } }
I20260620 23:08:06.599835 25936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 23:08:06 dist-test-slave-7g8g krb5kdc[25700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996885, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
I20260620 23:08:06.620980 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.590449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:39091 (local address 127.0.0.1:46010)
0620 23:08:06.590797 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:06.590818 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:06.590951 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:06.591840 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:06.591846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:06.592416 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:06.592630 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.592644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.594260 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.594266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.595836 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:06.595853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.601931 (+ 6078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.601937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:06.602175 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.602994 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:06.603054 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:06.607248 (+ 4194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:06.614133 (+ 6885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.614139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.614172 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.615773 (+ 1601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.616619 (+ 846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:06.616624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:06.616627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:06.616914 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:06.618306 (+ 1392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:06.618311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:06.618639 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 23:08:06.618847 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 23:08:06.619398 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260620 23:08:06.622529 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:06.591185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.12.2.124:39091)
0620 23:08:06.591350 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:06.591356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:06.591375 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:08:06.591431 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:06.591437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:06.591510 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:06.591599 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:06.592834 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.594028 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.600980 (+ 6952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:06.601771 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:06.601887 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:06.607503 (+ 5616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:06.607509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:06.607514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:06.607550 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:06.613874 (+ 6324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.616012 (+ 2138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.616017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.616020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.616344 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:06.617193 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:06.617198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:06.617201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:06.618093 (+ 892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:06.619731 (+ 1638us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:06.620354 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 23:08:06.621648 (+ 1294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:08:06.630285 25854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c: Bootstrap starting.
I20260620 23:08:06.636545 25854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:06.636907 25971 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:08:06.638664 25971 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:08:06.648651 25854 log.cc:824] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:06.652554 25971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924: Generated new TSK 0
I20260620 23:08:06.654048 25971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:08:06.655938 25854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c: No bootstrap required, opened a new log
I20260620 23:08:06.666857 25854 raft_consensus.cc:348] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } }
I20260620 23:08:06.667347 25854 raft_consensus.cc:374] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:06.667461 25854 raft_consensus.cc:729] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e477ee440bc94f71898ecf02474b070c, State: Initialized, Role: FOLLOWER
I20260620 23:08:06.668116 25854 consensus_queue.cc:260] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } }
I20260620 23:08:06.716315 25979 sys_catalog.cc:455] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } } }
I20260620 23:08:06.717504 25979 sys_catalog.cc:458] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:08:06.718577 25829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb56139364f44f35bfcaea5daa4e2924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e477ee440bc94f71898ecf02474b070c" is_pre_election: true
I20260620 23:08:06.719197 25829 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb56139364f44f35bfcaea5daa4e2924 in term 0.
I20260620 23:08:06.718921 25828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb56139364f44f35bfcaea5daa4e2924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e477ee440bc94f71898ecf02474b070c"
I20260620 23:08:06.719424 25828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:06.720341 25854 sys_catalog.cc:564] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:08:06.724972 25828 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb56139364f44f35bfcaea5daa4e2924 in term 1.
I20260620 23:08:06.725747 25827 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [term 1 FOLLOWER]: Refusing update from remote peer cb56139364f44f35bfcaea5daa4e2924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:06.727900 25969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:06.779578 25979 sys_catalog.cc:455] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb56139364f44f35bfcaea5daa4e2924. Latest consensus state: current_term: 1 leader_uuid: "cb56139364f44f35bfcaea5daa4e2924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } } }
I20260620 23:08:06.779961 25979 sys_catalog.cc:458] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:08:06.780321 25988 sys_catalog.cc:455] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb56139364f44f35bfcaea5daa4e2924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } } }
I20260620 23:08:06.780689 25988 sys_catalog.cc:458] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:08:06.805770 25999 catalog_manager.cc:1259] Loaded cluster ID: 1922bdf9f54a42f09981f2c24a6512cf
I20260620 23:08:06.805871 25999 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c: loading cluster ID for follower catalog manager: success
I20260620 23:08:06.809620 25999 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c: acquiring CA information for follower catalog manager: success
I20260620 23:08:06.812139 25999 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e477ee440bc94f71898ecf02474b070c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:08:07.216843 25969 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb56139364f44f35bfcaea5daa4e2924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c1b242306b477581bfa02677256db0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 38115 } } peers { permanent_uuid: "e477ee440bc94f71898ecf02474b070c" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 34193 } } peers { permanent_uuid: "cb56139364f44f35bfcaea5daa4e2924" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 39091 } } }
I20260620 23:08:07.217211 25969 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb56139364f44f35bfcaea5daa4e2924 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:07.455637 25949 catalog_manager.cc:1259] Loaded cluster ID: 1922bdf9f54a42f09981f2c24a6512cf
I20260620 23:08:07.455739 25949 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0: loading cluster ID for follower catalog manager: success
I20260620 23:08:07.458236 25949 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0: acquiring CA information for follower catalog manager: success
I20260620 23:08:07.459934 25949 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 18c1b242306b477581bfa02677256db0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:08:07.809062 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 25713
I20260620 23:08:07.810859 25775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:07.918483 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 25713
I20260620 23:08:07.935245 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 25787
I20260620 23:08:07.936409 25849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:08.041709 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 25787
W20260620 23:08:08.057739 25879 connection.cc:535] client connection to 127.12.2.125:34193 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:08:08.058068 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 25861
W20260620 23:08:08.058037 25879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:08:08.059306 25923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:08.164820 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:08.059200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:34193 (local address 127.0.0.1:45434)
0620 23:08:08.164444 (+105244us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:08.164602 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105119,"thread_start_us":179,"threads_started":1}
I20260620 23:08:08.164858 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 25861
2026-06-20T23:08:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4377 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:08:08 dist-test-slave-7g8g krb5kdc[26011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:08:08 dist-test-slave-7g8g krb5kdc[26011](info): set up 2 sockets
Jun 20 23:08:08 dist-test-slave-7g8g krb5kdc[26011](info): commencing operation
krb5kdc: starting...
W20260620 23:08:09.171180 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:08:09 dist-test-slave-7g8g krb5kdc[26011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:08:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:08:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:08:09.387573 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:35867
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:35867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:09.705420 26024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:09.705734 26024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:09.705873 26024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:09.713158 26024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:08:09.713279 26024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:09.713361 26024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:09.713442 26024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:08:09.713516 26024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:08:09.723440 26024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:35867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:35867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:09.724713 26024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:09.725931 26024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:09.736934 26029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:09.736963 26030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:09.736894 26032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:09.737675 26024 server_base.cc:1034] running on GCE node
I20260620 23:08:09.738368 26024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:09.739729 26024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:09.740927 26024 hybrid_clock.cc:648] HybridClock initialized: now 1781996889740873 us; error 46 us; skew 500 ppm
I20260620 23:08:09.741245 26024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:09 dist-test-slave-7g8g krb5kdc[26011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996889, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:09.744148 26024 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:08:09.745574 26024 webserver.cc:460] Webserver started at http://127.12.2.126:44467/ using document root <none> and password file <none>
I20260620 23:08:09.746235 26024 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:09.746351 26024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:09.746642 26024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:09.749537 26024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "adc797c8c572408d83b1f7e94c35c286"
format_stamp: "Formatted at 2026-06-20 23:08:09 on dist-test-slave-7g8g"
server_key: "277dc7f477818e7816d4ff30187e94f3"
server_key_iv: "5c0f4b5fb763af8de2a0dd6426d9f568"
server_key_version: "encryptionkey@0"
I20260620 23:08:09.750252 26024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "adc797c8c572408d83b1f7e94c35c286"
format_stamp: "Formatted at 2026-06-20 23:08:09 on dist-test-slave-7g8g"
server_key: "277dc7f477818e7816d4ff30187e94f3"
server_key_iv: "5c0f4b5fb763af8de2a0dd6426d9f568"
server_key_version: "encryptionkey@0"
I20260620 23:08:09.755285 26024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:08:09.758641 26039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:09.759953 26024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:08:09.760110 26024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "adc797c8c572408d83b1f7e94c35c286"
format_stamp: "Formatted at 2026-06-20 23:08:09 on dist-test-slave-7g8g"
server_key: "277dc7f477818e7816d4ff30187e94f3"
server_key_iv: "5c0f4b5fb763af8de2a0dd6426d9f568"
server_key_version: "encryptionkey@0"
I20260620 23:08:09.760288 26024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:09.785037 26024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:09.788825 26024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:09.789067 26024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:09.805769 26024 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:35867
I20260620 23:08:09.805799 26090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:35867 every 8 connection(s)
I20260620 23:08:09.807037 26024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:08:09.810623 26091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:09.816669 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 26024
I20260620 23:08:09.816884 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:08:09.817680 12297 external_mini_cluster.cc:1428] Setting key 0d57edde5daba4523cfed51a3254bed9
I20260620 23:08:09.821348 26091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286: Bootstrap starting.
I20260620 23:08:09.824843 26091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:09.825868 26091 log.cc:824] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:09.828692 26091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286: No bootstrap required, opened a new log
Jun 20 23:08:09 dist-test-slave-7g8g krb5kdc[26011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:09.838421 26091 raft_consensus.cc:348] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc797c8c572408d83b1f7e94c35c286" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35867 } }
I20260620 23:08:09.838974 26091 raft_consensus.cc:374] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:09.839083 26091 raft_consensus.cc:729] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc797c8c572408d83b1f7e94c35c286, State: Initialized, Role: FOLLOWER
I20260620 23:08:09.839730 26091 consensus_queue.cc:260] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc797c8c572408d83b1f7e94c35c286" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35867 } }
I20260620 23:08:09.840018 26091 raft_consensus.cc:388] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:08:09.840160 26091 raft_consensus.cc:482] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:08:09.840349 26091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:09.840788 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:09.819511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.12.2.126:35867)
0620 23:08:09.820257 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:09.820268 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:09.820309 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:08:09.821968 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:09.821989 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:09.822355 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:09.822673 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:09.824397 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:09.825724 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:09.827265 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:09.827913 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:09.828037 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:09.832569 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:09.832578 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:09.832596 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:09.832647 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:09.836440 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:09.837101 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:09.837109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:09.837118 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:09.837326 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:09.837650 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:09.837655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:09.837659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:09.838338 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:09.838626 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:09.838969 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 23:08:09.839423 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":328,"threads_started":1}
I20260620 23:08:09.842743 26091 raft_consensus.cc:504] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc797c8c572408d83b1f7e94c35c286" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35867 } }
I20260620 23:08:09.843232 26091 leader_election.cc:302] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adc797c8c572408d83b1f7e94c35c286; no voters:
I20260620 23:08:09.843804 26091 leader_election.cc:288] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:08:09.843899 26096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:08:09.844476 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:08:09.845163 26096 raft_consensus.cc:686] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [term 1 LEADER]: Becoming Leader. State: Replica: adc797c8c572408d83b1f7e94c35c286, State: Running, Role: LEADER
I20260620 23:08:09.845638 26096 consensus_queue.cc:237] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc797c8c572408d83b1f7e94c35c286" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35867 } }
I20260620 23:08:09.845906 26091 sys_catalog.cc:564] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:08:09.848013 26098 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc797c8c572408d83b1f7e94c35c286. Latest consensus state: current_term: 1 leader_uuid: "adc797c8c572408d83b1f7e94c35c286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc797c8c572408d83b1f7e94c35c286" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35867 } } }
I20260620 23:08:09.848399 26098 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:09.848615 26097 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adc797c8c572408d83b1f7e94c35c286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc797c8c572408d83b1f7e94c35c286" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35867 } } }
I20260620 23:08:09.848906 26097 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:09.849336 26104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:08:09.858505 26104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:08:09.867403 26104 catalog_manager.cc:1347] Generated new cluster ID: 2901e03cc89748078ad84610ca342182
I20260620 23:08:09.867506 26104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:08:09.880435 26104 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:08:09.881654 26104 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:08:09.888275 26104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adc797c8c572408d83b1f7e94c35c286: Generated new TSK 0
I20260620 23:08:09.889037 26104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:08:09.904955 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 26024
I20260620 23:08:09.906025 26086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:09.997159 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 26024
2026-06-20T23:08:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1829 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:08:10 dist-test-slave-7g8g krb5kdc[26119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:08:10 dist-test-slave-7g8g krb5kdc[26119](info): set up 2 sockets
Jun 20 23:08:10 dist-test-slave-7g8g krb5kdc[26119](info): commencing operation
krb5kdc: starting...
W20260620 23:08:10.995232 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:08:11 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:08:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:08:11.204214 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:36055
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:36055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:11.515334 26132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:11.515607 26132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:11.515697 26132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:11.522586 26132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:08:11.522683 26132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:11.522734 26132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:11.522789 26132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:08:11.522830 26132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:08:11.532238 26132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:36055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:36055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:11.533504 26132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:11.534714 26132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:11.545397 26140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:11.545360 26138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:11.545506 26137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:11.546310 26132 server_base.cc:1034] running on GCE node
I20260620 23:08:11.547091 26132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:11.548386 26132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:11.549616 26132 hybrid_clock.cc:648] HybridClock initialized: now 1781996891549563 us; error 36 us; skew 500 ppm
I20260620 23:08:11.549978 26132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:11 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996891, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:11.555819 26132 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:08:11.557245 26132 webserver.cc:460] Webserver started at http://127.12.2.126:41835/ using document root <none> and password file <none>
I20260620 23:08:11.557884 26132 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:11.557998 26132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:11.558256 26132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:11.561245 26132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "b26c4faf5ad84fcf901967b09261e0e9"
format_stamp: "Formatted at 2026-06-20 23:08:11 on dist-test-slave-7g8g"
server_key: "8616e5425536cbf0897fc64f906cc396"
server_key_iv: "bcca3427a183a5f31e9bfeed473c3594"
server_key_version: "encryptionkey@0"
I20260620 23:08:11.561985 26132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "b26c4faf5ad84fcf901967b09261e0e9"
format_stamp: "Formatted at 2026-06-20 23:08:11 on dist-test-slave-7g8g"
server_key: "8616e5425536cbf0897fc64f906cc396"
server_key_iv: "bcca3427a183a5f31e9bfeed473c3594"
server_key_version: "encryptionkey@0"
I20260620 23:08:11.566916 26132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:08:11.570278 26147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:11.571615 26132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:08:11.571759 26132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "b26c4faf5ad84fcf901967b09261e0e9"
format_stamp: "Formatted at 2026-06-20 23:08:11 on dist-test-slave-7g8g"
server_key: "8616e5425536cbf0897fc64f906cc396"
server_key_iv: "bcca3427a183a5f31e9bfeed473c3594"
server_key_version: "encryptionkey@0"
I20260620 23:08:11.571921 26132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:11.594764 26132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:11.598737 26132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:11.598994 26132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:11.615784 26132 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:36055
I20260620 23:08:11.615813 26198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:36055 every 8 connection(s)
I20260620 23:08:11.617077 26132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:08:11.620666 26199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:11.624186 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 26132
I20260620 23:08:11.624365 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:08:11.624972 12297 external_mini_cluster.cc:1428] Setting key ac3ccf687f1ce1daa355ec65ba46e9bc
I20260620 23:08:11.632244 26199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Bootstrap starting.
I20260620 23:08:11.635743 26199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:11.636919 26199 log.cc:824] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:11.639781 26199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: No bootstrap required, opened a new log
Jun 20 23:08:11 dist-test-slave-7g8g krb5kdc[26119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:11.649515 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:11.626818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.12.2.126:36055)
0620 23:08:11.627579 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:11.627595 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:11.627643 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:08:11.630558 (+ 2915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:11.630588 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:11.631094 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:11.631512 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:11.632833 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:11.634792 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:11.636149 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:11.637142 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:11.637328 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:11.641225 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:11.641235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:11.641253 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:11.641298 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:11.645161 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:11.645859 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:11.645866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:11.645878 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:11.646074 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:11.646460 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:11.646465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:11.646469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:11.647165 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:11.647481 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:11.647759 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 23:08:11.648213 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":278,"threads_started":1}
I20260620 23:08:11.649686 26199 raft_consensus.cc:348] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:11.650096 26199 raft_consensus.cc:374] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:11.650190 26199 raft_consensus.cc:729] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26c4faf5ad84fcf901967b09261e0e9, State: Initialized, Role: FOLLOWER
I20260620 23:08:11.650735 26199 consensus_queue.cc:260] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:11.650965 26199 raft_consensus.cc:388] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:08:11.651100 26199 raft_consensus.cc:482] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:08:11.651261 26199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:11.653434 26199 raft_consensus.cc:504] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:11.653892 26199 leader_election.cc:302] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b26c4faf5ad84fcf901967b09261e0e9; no voters:
I20260620 23:08:11.654410 26199 leader_election.cc:288] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:08:11.654568 26204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:08:11.655295 26204 raft_consensus.cc:686] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 1 LEADER]: Becoming Leader. State: Replica: b26c4faf5ad84fcf901967b09261e0e9, State: Running, Role: LEADER
I20260620 23:08:11.655949 26204 consensus_queue.cc:237] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:11.656303 26199 sys_catalog.cc:564] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:08:11.658849 26206 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b26c4faf5ad84fcf901967b09261e0e9. Latest consensus state: current_term: 1 leader_uuid: "b26c4faf5ad84fcf901967b09261e0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } } }
I20260620 23:08:11.659253 26206 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:11.659164 26205 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b26c4faf5ad84fcf901967b09261e0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } } }
I20260620 23:08:11.659487 26205 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:11.663849 26212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:08:11.668998 26212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:08:11.677472 26212 catalog_manager.cc:1347] Generated new cluster ID: feffd4ee8df845c1aeb8e058885901aa
I20260620 23:08:11.677591 26212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:08:11.703459 26212 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:08:11.704784 26212 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:08:11.713178 26212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Generated new TSK 0
I20260620 23:08:11.714042 26212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:08:11.784217 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36055
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:12.099684 26225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:12.099948 26225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:12.100049 26225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:12.107255 26225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:12.107354 26225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:12.107475 26225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:08:12.116746 26225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:12.117936 26225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:12.119199 26225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:12.131378 26230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:12.131603 26231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:12.132121 26233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:12.132829 26225 server_base.cc:1034] running on GCE node
I20260620 23:08:12.133428 26225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:12.134378 26225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:12.135584 26225 hybrid_clock.cc:648] HybridClock initialized: now 1781996892135524 us; error 49 us; skew 500 ppm
I20260620 23:08:12.135883 26225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:12 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996892, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:12.138986 26225 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:08:12.140405 26225 webserver.cc:460] Webserver started at http://127.12.2.65:43467/ using document root <none> and password file <none>
I20260620 23:08:12.141161 26225 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:12.141294 26225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:12.141553 26225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:12.144404 26225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "1b484a0c2cf44b0a8e3f52ef9291a968"
format_stamp: "Formatted at 2026-06-20 23:08:12 on dist-test-slave-7g8g"
server_key: "12af4a1da52b8fe7228c2eb5878a5b59"
server_key_iv: "e0fffb4b1adf40a818b433968afab846"
server_key_version: "encryptionkey@0"
I20260620 23:08:12.145112 26225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "1b484a0c2cf44b0a8e3f52ef9291a968"
format_stamp: "Formatted at 2026-06-20 23:08:12 on dist-test-slave-7g8g"
server_key: "12af4a1da52b8fe7228c2eb5878a5b59"
server_key_iv: "e0fffb4b1adf40a818b433968afab846"
server_key_version: "encryptionkey@0"
I20260620 23:08:12.149919 26225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:08:12.153194 26240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:12.154474 26225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:08:12.154606 26225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "1b484a0c2cf44b0a8e3f52ef9291a968"
format_stamp: "Formatted at 2026-06-20 23:08:12 on dist-test-slave-7g8g"
server_key: "12af4a1da52b8fe7228c2eb5878a5b59"
server_key_iv: "e0fffb4b1adf40a818b433968afab846"
server_key_version: "encryptionkey@0"
I20260620 23:08:12.154776 26225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:12.186537 26225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:12.190843 26225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:12.191193 26225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:12.192103 26225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:12.193338 26225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:12.193406 26225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:12.193498 26225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:12.193538 26225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:12.222265 26225 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:36499
I20260620 23:08:12.222303 26352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:36499 every 8 connection(s)
I20260620 23:08:12.223517 26225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:08:12.227979 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 26225
I20260620 23:08:12.228183 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:08:12.228936 12297 external_mini_cluster.cc:1428] Setting key 388560378f01a5cd08a6049fada07173
Jun 20 23:08:12 dist-test-slave-7g8g krb5kdc[26119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996892, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:12.244123 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:12.225823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:37601 (local address 127.12.2.126:36055)
0620 23:08:12.226234 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:12.226238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:12.227146 (+ 908us) server_negotiation.cc:407] Connection header received
0620 23:08:12.229482 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:12.229487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:12.229565 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:12.229685 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:12.231336 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:12.232059 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:12.234178 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:12.234617 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:12.234722 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:12.238533 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:12.238539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:12.238542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:12.238574 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:12.241361 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:12.242007 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:12.242011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:12.242015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:12.242148 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:12.242498 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:12.242502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:12.242505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:12.242806 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:12.242925 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:12.243566 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 23:08:12.243773 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":195,"threads_started":1}
I20260620 23:08:12.244838 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:12.226334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:37601)
0620 23:08:12.227011 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:12.227049 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:12.229195 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:12.229910 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:12.229922 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:12.230418 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:12.231172 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:12.231197 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:12.232226 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:12.232230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:12.234003 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:12.234017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:12.234826 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:12.234832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:12.235140 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:12.235836 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:12.235866 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:12.238302 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:12.241495 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:12.241501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:12.241513 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:12.241899 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:12.242232 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:12.242235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:12.242238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:12.242394 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:12.242906 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:12.242912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:12.243231 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 23:08:12.243439 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 23:08:12.243796 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":268,"threads_started":1}
I20260620 23:08:12.246325 26353 heartbeater.cc:347] Connected to a master server at 127.12.2.126:36055
I20260620 23:08:12.246592 26353 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:12.247301 26353 heartbeater.cc:511] Master 127.12.2.126:36055 requested a full tablet report, sending...
I20260620 23:08:12.249334 26164 ts_manager.cc:194] Registered new tserver with Master: 1b484a0c2cf44b0a8e3f52ef9291a968 (127.12.2.65:36499)
I20260620 23:08:12.250752 26164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:37601
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:08:12.277540 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36055
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:12.585443 26359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:12.585764 26359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:12.585907 26359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:12.592891 26359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:12.592986 26359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:12.593137 26359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:08:12.602586 26359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:12.603798 26359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:12.605091 26359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:12.618016 26364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:12.618077 26365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:12.619518 26367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:12.620082 26359 server_base.cc:1034] running on GCE node
I20260620 23:08:12.620648 26359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:12.621559 26359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:12.622813 26359 hybrid_clock.cc:648] HybridClock initialized: now 1781996892622756 us; error 48 us; skew 500 ppm
I20260620 23:08:12.623126 26359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:12 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996892, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:12.626332 26359 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:08:12.628029 26359 webserver.cc:460] Webserver started at http://127.12.2.66:33509/ using document root <none> and password file <none>
I20260620 23:08:12.628885 26359 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:12.629025 26359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:12.629359 26359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:12.633231 26359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee"
format_stamp: "Formatted at 2026-06-20 23:08:12 on dist-test-slave-7g8g"
server_key: "2688aad9ef9ac6795b37a6f8937aad92"
server_key_iv: "3760f7740b9ee20dcd56a682e4247bef"
server_key_version: "encryptionkey@0"
I20260620 23:08:12.634064 26359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee"
format_stamp: "Formatted at 2026-06-20 23:08:12 on dist-test-slave-7g8g"
server_key: "2688aad9ef9ac6795b37a6f8937aad92"
server_key_iv: "3760f7740b9ee20dcd56a682e4247bef"
server_key_version: "encryptionkey@0"
I20260620 23:08:12.638859 26359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 23:08:12.642194 26374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:12.643558 26359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:08:12.643690 26359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee"
format_stamp: "Formatted at 2026-06-20 23:08:12 on dist-test-slave-7g8g"
server_key: "2688aad9ef9ac6795b37a6f8937aad92"
server_key_iv: "3760f7740b9ee20dcd56a682e4247bef"
server_key_version: "encryptionkey@0"
I20260620 23:08:12.643858 26359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:12.660575 26359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:12.664424 26359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:12.664672 26359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:12.665570 26359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:12.667006 26359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:12.667078 26359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:12.667187 26359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:12.667234 26359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:12.694540 26359 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:38259
I20260620 23:08:12.694586 26486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:38259 every 8 connection(s)
I20260620 23:08:12.695816 26359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:08:12.697609 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 26359
I20260620 23:08:12.697767 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:08:12.698308 12297 external_mini_cluster.cc:1428] Setting key 0ca280f3c5b0ec53711d8cd2b95087b8
Jun 20 23:08:12 dist-test-slave-7g8g krb5kdc[26119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996892, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:12.716491 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:12.699054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:34561 (local address 127.12.2.126:36055)
0620 23:08:12.700139 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:12.700150 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:12.700408 (+ 258us) server_negotiation.cc:407] Connection header received
0620 23:08:12.702394 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:12.702401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:12.702481 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:12.702590 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:12.704151 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:12.705156 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:12.706830 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:12.707411 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:12.707535 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:12.711283 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:12.711288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:12.711291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:12.711323 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:12.713662 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:12.714332 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:12.714337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:12.714339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:12.714457 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:12.714854 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:12.714857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:12.714859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:12.715150 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:12.715279 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:12.715929 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 23:08:12.716158 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260620 23:08:12.717413 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:12.699288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:34561)
0620 23:08:12.700267 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:12.700311 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:12.702108 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:12.702782 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:12.702790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:12.703371 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:12.703983 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:12.704000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:12.705322 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:12.705327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:12.706691 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:12.706704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:12.707648 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:12.707656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:12.707963 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:12.708649 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:12.708679 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:12.711074 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:12.713785 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:12.713794 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:12.713811 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:12.714213 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:12.714562 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:12.714567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:12.714571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:12.714746 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:12.715263 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:12.715270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:12.715591 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 23:08:12.715809 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 23:08:12.716311 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":459,"threads_started":1}
I20260620 23:08:12.718802 26487 heartbeater.cc:347] Connected to a master server at 127.12.2.126:36055
I20260620 23:08:12.719108 26487 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
I20260620 23:08:12.719790 26487 heartbeater.cc:511] Master 127.12.2.126:36055 requested a full tablet report, sending...
I20260620 23:08:12.721243 26164 ts_manager.cc:194] Registered new tserver with Master: 83c3f45d312d41ee98c8c5bf4dbfb7ee (127.12.2.66:38259)
I20260620 23:08:12.722102 26164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:34561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:08:12.745833 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36055
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:13.052338 26492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:13.052605 26492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:13.052696 26492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:13.059811 26492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:13.059913 26492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:13.060037 26492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:08:13.069564 26492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:13.070776 26492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:13.072075 26492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:13.084262 26500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:13.084340 26497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:13.084372 26498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:13.084885 26492 server_base.cc:1034] running on GCE node
I20260620 23:08:13.085558 26492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:13.086438 26492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:13.087647 26492 hybrid_clock.cc:648] HybridClock initialized: now 1781996893087546 us; error 102 us; skew 500 ppm
I20260620 23:08:13.087955 26492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:13 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996893, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:13.090898 26492 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:08:13.092341 26492 webserver.cc:460] Webserver started at http://127.12.2.67:38433/ using document root <none> and password file <none>
I20260620 23:08:13.092953 26492 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:13.093057 26492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:13.093297 26492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:13.096232 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a"
format_stamp: "Formatted at 2026-06-20 23:08:13 on dist-test-slave-7g8g"
server_key: "b97be12e711a60e28eab4d6b6e6c3ee7"
server_key_iv: "4df8e949d9170130e0535e3bcde72cb5"
server_key_version: "encryptionkey@0"
I20260620 23:08:13.096931 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a"
format_stamp: "Formatted at 2026-06-20 23:08:13 on dist-test-slave-7g8g"
server_key: "b97be12e711a60e28eab4d6b6e6c3ee7"
server_key_iv: "4df8e949d9170130e0535e3bcde72cb5"
server_key_version: "encryptionkey@0"
I20260620 23:08:13.101740 26492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:08:13.105139 26507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:13.106462 26492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:08:13.106598 26492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a"
format_stamp: "Formatted at 2026-06-20 23:08:13 on dist-test-slave-7g8g"
server_key: "b97be12e711a60e28eab4d6b6e6c3ee7"
server_key_iv: "4df8e949d9170130e0535e3bcde72cb5"
server_key_version: "encryptionkey@0"
I20260620 23:08:13.106761 26492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:13.128780 26492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:13.132560 26492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:13.132823 26492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:13.133767 26492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:13.135095 26492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:13.135169 26492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:13.135258 26492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:13.135303 26492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:13.163784 26492 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:34451
I20260620 23:08:13.163820 26619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:34451 every 8 connection(s)
I20260620 23:08:13.165117 26492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:08:13.166523 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 26492
I20260620 23:08:13.166702 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:08:13.167232 12297 external_mini_cluster.cc:1428] Setting key 9351cb045b304ac8a4816741444614cd
Jun 20 23:08:13 dist-test-slave-7g8g krb5kdc[26119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996893, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:13.183813 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.167681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:39839 (local address 127.12.2.126:36055)
0620 23:08:13.167887 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:13.167893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:13.169142 (+ 1249us) server_negotiation.cc:407] Connection header received
0620 23:08:13.170964 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:13.170969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:13.171062 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:13.171153 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:13.172521 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.173236 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.174788 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.175275 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.175375 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.178656 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:13.178660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:13.178663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:13.178693 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:13.181245 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:13.181980 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:13.181985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:13.181989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:13.182127 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:13.182466 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:13.182468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:13.182470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:13.182712 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:13.182807 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:13.183423 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 23:08:13.183571 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:08:13.184728 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.168221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:39839)
0620 23:08:13.169013 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:13.169058 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:13.170698 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:13.171333 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:13.171341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:13.171864 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:13.172399 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.172414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.173351 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.173354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.174590 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.174601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.175488 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.175493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.175764 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.176405 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:13.176425 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:13.178476 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:13.181376 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:13.181384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:13.181398 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:13.181877 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:13.182207 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:13.182211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:13.182213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:13.182391 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:13.182806 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:13.182811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:13.183131 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 23:08:13.183348 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 23:08:13.183689 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":324,"threads_started":1}
I20260620 23:08:13.185856 26620 heartbeater.cc:347] Connected to a master server at 127.12.2.126:36055
I20260620 23:08:13.186116 26620 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:13.186764 26620 heartbeater.cc:511] Master 127.12.2.126:36055 requested a full tablet report, sending...
I20260620 23:08:13.188079 26164 ts_manager.cc:194] Registered new tserver with Master: 6ee80df0b37c4d8aa6f4e70d9ee0943a (127.12.2.67:34451)
I20260620 23:08:13.188848 26164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:39839
I20260620 23:08:13.193993 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:08:13.209739 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.198258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.12.2.126:36055)
0620 23:08:13.198401 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:13.198405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:13.198801 (+ 396us) server_negotiation.cc:407] Connection header received
0620 23:08:13.198919 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:13.198923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:13.199012 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:13.199090 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:13.200372 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.201097 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.202420 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.202812 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.202900 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.205166 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:13.205171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:13.205174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:13.205202 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:13.207599 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:13.208155 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:13.208159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:13.208162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:13.208282 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:13.208589 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:13.208592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:13.208594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:13.208881 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:13.209055 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:13.209309 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 23:08:13.209453 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:08:13.213457 26164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:08:13.220099 26164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:08:13.253760 26353 heartbeater.cc:503] Master 127.12.2.126:36055 was elected leader, sending a full tablet report...
I20260620 23:08:13.256745 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.232933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.12.2.66:38259)
0620 23:08:13.248631 (+ 15698us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:13.248638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:13.249282 (+ 644us) server_negotiation.cc:407] Connection header received
0620 23:08:13.249373 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:13.249378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:13.249534 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:13.249683 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:13.250340 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.251802 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.254259 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.255586 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.255700 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.255891 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:13.256387 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 23:08:13.256532 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15466,"thread_start_us":16183,"threads_started":1}
I20260620 23:08:13.256846 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.232108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.12.2.67:34451)
0620 23:08:13.247999 (+ 15891us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:13.248009 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:13.248045 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:08:13.248155 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:13.248163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:13.248418 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:13.249750 (+ 1332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:13.250014 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.251526 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.254160 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.255549 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.256045 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.256339 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:13.256405 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 23:08:13.256549 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15736,"thread_start_us":15580,"threads_started":1}
I20260620 23:08:13.256877 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.231812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:34451 (local address 127.0.0.1:39496)
0620 23:08:13.242666 (+ 10854us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:13.242713 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:13.242877 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:13.249092 (+ 6215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:13.249099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:13.249130 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:13.249555 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.249567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.251668 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.251677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.253914 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.253928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.255816 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.255823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.255998 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.256125 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 23:08:13.256249 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 23:08:13.256466 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10617,"mutex_wait_us":51,"thread_start_us":16965,"threads_started":1}
I20260620 23:08:13.257596 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.232364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:38259 (local address 127.0.0.1:38654)
0620 23:08:13.249028 (+ 16664us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:13.249044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:13.249160 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:13.249698 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:13.249702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:13.249733 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:13.250152 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.250165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.251958 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.251963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.254106 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.254118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.255779 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.255785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.256051 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.256170 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 23:08:13.256248 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:08:13.257347 (+ 1099us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16541,"mutex_wait_us":999,"thread_start_us":15704,"threads_started":1}
I20260620 23:08:13.258383 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.232047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:36499 (local address 127.0.0.1:36248)
0620 23:08:13.248378 (+ 16331us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:13.248394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:13.248511 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:13.249380 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:13.249384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:13.249410 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:13.249827 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.249842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.251459 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.251464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.253368 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.253380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.257138 (+ 3758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.257144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.257853 (+ 709us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.257876 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:08:13.257949 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:08:13.258064 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16215,"thread_start_us":10257,"threads_started":1}
I20260620 23:08:13.259133 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.232542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.12.2.65:36499)
0620 23:08:13.247811 (+ 15269us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:13.247820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:13.248654 (+ 834us) server_negotiation.cc:407] Connection header received
0620 23:08:13.248765 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:13.248773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:13.248980 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:13.249164 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:13.250012 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.251311 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.253518 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.255974 (+ 2456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.256094 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.256305 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:13.258638 (+ 2333us) server_negotiation.cc:299] Negotiation successful
0620 23:08:13.258794 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15096,"thread_start_us":15256,"threads_started":1}
I20260620 23:08:13.261018 26555 tablet_service.cc:1467] Processing CreateTablet for tablet 5ff852fb3a0b4d5cb1aa1d634317e10f (DEFAULT_TABLE table=test-table [id=7445a3d5fde04340a3d15ff15581278c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:13.261834 26288 tablet_service.cc:1467] Processing CreateTablet for tablet 5ff852fb3a0b4d5cb1aa1d634317e10f (DEFAULT_TABLE table=test-table [id=7445a3d5fde04340a3d15ff15581278c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:13.262477 26555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ff852fb3a0b4d5cb1aa1d634317e10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:13.262913 26288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ff852fb3a0b4d5cb1aa1d634317e10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:13.262550 26422 tablet_service.cc:1467] Processing CreateTablet for tablet 5ff852fb3a0b4d5cb1aa1d634317e10f (DEFAULT_TABLE table=test-table [id=7445a3d5fde04340a3d15ff15581278c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:13.263842 26422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ff852fb3a0b4d5cb1aa1d634317e10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:13.273922 26638 tablet_bootstrap.cc:492] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968: Bootstrap starting.
I20260620 23:08:13.276374 26639 tablet_bootstrap.cc:492] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Bootstrap starting.
I20260620 23:08:13.278570 26638 tablet_bootstrap.cc:654] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:13.279784 26639 tablet_bootstrap.cc:654] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:13.279901 26638 log.cc:824] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:13.280982 26639 log.cc:824] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:13.282220 26640 tablet_bootstrap.cc:492] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Bootstrap starting.
I20260620 23:08:13.282968 26638 tablet_bootstrap.cc:492] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968: No bootstrap required, opened a new log
I20260620 23:08:13.283242 26638 ts_tablet_manager.cc:1397] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260620 23:08:13.283864 26639 tablet_bootstrap.cc:492] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: No bootstrap required, opened a new log
I20260620 23:08:13.284129 26639 ts_tablet_manager.cc:1397] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 23:08:13.284858 26640 tablet_bootstrap.cc:654] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:13.285933 26640 log.cc:824] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:13.288298 26640 tablet_bootstrap.cc:492] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: No bootstrap required, opened a new log
I20260620 23:08:13.288514 26640 ts_tablet_manager.cc:1397] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 23:08:13.294014 26638 raft_consensus.cc:348] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } }
I20260620 23:08:13.294023 26639 raft_consensus.cc:348] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } }
I20260620 23:08:13.294468 26639 raft_consensus.cc:374] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:13.294503 26638 raft_consensus.cc:374] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:13.294574 26639 raft_consensus.cc:729] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee80df0b37c4d8aa6f4e70d9ee0943a, State: Initialized, Role: FOLLOWER
I20260620 23:08:13.294610 26638 raft_consensus.cc:729] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b484a0c2cf44b0a8e3f52ef9291a968, State: Initialized, Role: FOLLOWER
I20260620 23:08:13.295231 26638 consensus_queue.cc:260] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } }
I20260620 23:08:13.295207 26640 raft_consensus.cc:348] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } }
I20260620 23:08:13.295656 26640 raft_consensus.cc:374] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:13.295743 26640 raft_consensus.cc:729] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c3f45d312d41ee98c8c5bf4dbfb7ee, State: Initialized, Role: FOLLOWER
I20260620 23:08:13.296363 26640 consensus_queue.cc:260] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } }
I20260620 23:08:13.299609 26639 consensus_queue.cc:260] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } }
I20260620 23:08:13.303407 26638 ts_tablet_manager.cc:1428] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968: Time spent starting tablet: real 0.020s user 0.009s sys 0.010s
I20260620 23:08:13.303469 26620 heartbeater.cc:503] Master 127.12.2.126:36055 was elected leader, sending a full tablet report...
I20260620 23:08:13.304345 26639 ts_tablet_manager.cc:1428] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Time spent starting tablet: real 0.020s user 0.015s sys 0.000s
I20260620 23:08:13.305137 26487 heartbeater.cc:503] Master 127.12.2.126:36055 was elected leader, sending a full tablet report...
I20260620 23:08:13.306181 26640 ts_tablet_manager.cc:1428] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Time spent starting tablet: real 0.018s user 0.010s sys 0.005s
I20260620 23:08:13.403625 26646 raft_consensus.cc:482] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:08:13.403896 26646 raft_consensus.cc:504] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } }
I20260620 23:08:13.405534 26646 leader_election.cc:288] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b484a0c2cf44b0a8e3f52ef9291a968 (127.12.2.65:36499), 83c3f45d312d41ee98c8c5bf4dbfb7ee (127.12.2.66:38259)
I20260620 23:08:13.410981 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.405903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:57913 (local address 127.12.2.65:36499)
0620 23:08:13.406087 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:13.406093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:13.406167 (+ 74us) server_negotiation.cc:407] Connection header received
0620 23:08:13.406437 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:13.406442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:13.406523 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:13.406611 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:13.407072 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.407779 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.409320 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.410132 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.410246 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.410343 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:13.410596 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 23:08:13.410715 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:08:13.411026 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.405763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:36499 (local address 127.12.2.67:57913)
0620 23:08:13.406073 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:13.406093 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:13.406233 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:13.406661 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:13.406666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:13.406692 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:13.406904 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.406913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.407904 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.407909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.409176 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.409187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.410281 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.410287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.410449 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.410473 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:08:13.410558 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:08:13.410711 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":22,"spinlock_wait_cycles":39680}
I20260620 23:08:13.411839 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.405998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:41721 (local address 127.12.2.66:38259)
0620 23:08:13.406200 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:13.406206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:13.406744 (+ 538us) server_negotiation.cc:407] Connection header received
0620 23:08:13.406950 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:13.406956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:13.407023 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:13.407105 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:13.407579 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.408269 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.409889 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.410762 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.410878 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.410996 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:13.411487 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 23:08:13.411608 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:08:13.411916 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.405886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:38259 (local address 127.12.2.67:41721)
0620 23:08:13.406644 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:13.406658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:13.406811 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:13.407122 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:13.407126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:13.407148 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:13.407444 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.407452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.408413 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.408418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.409757 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:13.409769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.410974 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.410980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:13.411317 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.411372 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 23:08:13.411445 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:08:13.411555 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":1,"thread_start_us":443,"threads_started":1}
I20260620 23:08:13.411813 26308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff852fb3a0b4d5cb1aa1d634317e10f" candidate_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" is_pre_election: true
I20260620 23:08:13.412328 26308 raft_consensus.cc:2454] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ee80df0b37c4d8aa6f4e70d9ee0943a in term 0.
I20260620 23:08:13.412710 26442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff852fb3a0b4d5cb1aa1d634317e10f" candidate_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" is_pre_election: true
I20260620 23:08:13.412914 26508 leader_election.cc:302] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b484a0c2cf44b0a8e3f52ef9291a968, 6ee80df0b37c4d8aa6f4e70d9ee0943a; no voters:
I20260620 23:08:13.413120 26442 raft_consensus.cc:2454] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ee80df0b37c4d8aa6f4e70d9ee0943a in term 0.
I20260620 23:08:13.413273 26646 raft_consensus.cc:2781] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:08:13.413414 26646 raft_consensus.cc:482] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:08:13.413508 26646 raft_consensus.cc:3037] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:13.415695 26646 raft_consensus.cc:504] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } }
I20260620 23:08:13.416225 26646 leader_election.cc:288] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [CANDIDATE]: Term 1 election: Requested vote from peers 1b484a0c2cf44b0a8e3f52ef9291a968 (127.12.2.65:36499), 83c3f45d312d41ee98c8c5bf4dbfb7ee (127.12.2.66:38259)
W20260620 23:08:13.416678 26621 tablet.cc:2369] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:08:13.416677 26308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff852fb3a0b4d5cb1aa1d634317e10f" candidate_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968"
I20260620 23:08:13.416682 26442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff852fb3a0b4d5cb1aa1d634317e10f" candidate_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee"
I20260620 23:08:13.416841 26308 raft_consensus.cc:3037] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:13.416857 26442 raft_consensus.cc:3037] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:13.418951 26308 raft_consensus.cc:2454] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ee80df0b37c4d8aa6f4e70d9ee0943a in term 1.
I20260620 23:08:13.419080 26442 raft_consensus.cc:2454] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ee80df0b37c4d8aa6f4e70d9ee0943a in term 1.
I20260620 23:08:13.419302 26508 leader_election.cc:302] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b484a0c2cf44b0a8e3f52ef9291a968, 6ee80df0b37c4d8aa6f4e70d9ee0943a; no voters:
I20260620 23:08:13.419551 26646 raft_consensus.cc:2781] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:08:13.419930 26646 raft_consensus.cc:686] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 1 LEADER]: Becoming Leader. State: Replica: 6ee80df0b37c4d8aa6f4e70d9ee0943a, State: Running, Role: LEADER
I20260620 23:08:13.420353 26646 consensus_queue.cc:237] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } }
I20260620 23:08:13.424443 26164 catalog_manager.cc:5495] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ee80df0b37c4d8aa6f4e70d9ee0943a (127.12.2.67). New cstate: current_term: 1 leader_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } health_report { overall_health: UNKNOWN } } }
I20260620 23:08:13.447212 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.442279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39508 (local address 127.12.2.67:34451)
0620 23:08:13.442462 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:13.442466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:13.442536 (+ 70us) server_negotiation.cc:407] Connection header received
0620 23:08:13.442686 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:13.442690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:13.442771 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:13.442883 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:08:13.443218 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.443944 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.445314 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.445687 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.445775 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.446192 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:08:13.446691 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:08:13.446794 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:13.446891 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 23:08:13.446980 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 23:08:13.447790 26488 tablet.cc:2369] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:08:13.455076 26442 raft_consensus.cc:1261] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 1 FOLLOWER]: Refusing update from remote peer 6ee80df0b37c4d8aa6f4e70d9ee0943a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:13.455109 26308 raft_consensus.cc:1261] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 1 FOLLOWER]: Refusing update from remote peer 6ee80df0b37c4d8aa6f4e70d9ee0943a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:13.456033 26646 consensus_queue.cc:1035] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:13.456308 26648 consensus_queue.cc:1035] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:13.466214 26653 mvcc.cc:205] Tried to move back new op lower bound from 7299059275582984192 to 7299059275452997632. Current Snapshot: MvccSnapshot[applied={T|T < 7299059275582984192}]
I20260620 23:08:13.480957 26163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:08:13.481220 26163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:08:13.484619 26163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Sending DeleteTablet for 3 replicas of tablet 5ff852fb3a0b4d5cb1aa1d634317e10f
I20260620 23:08:13.485709 26422 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ff852fb3a0b4d5cb1aa1d634317e10f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:13 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:38654
I20260620 23:08:13.485720 26555 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ff852fb3a0b4d5cb1aa1d634317e10f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:13 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:39496
I20260620 23:08:13.485795 26288 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ff852fb3a0b4d5cb1aa1d634317e10f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:13 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:36248
I20260620 23:08:13.488150 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 26225
I20260620 23:08:13.497987 26662 tablet_replica.cc:330] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968: stopping tablet replica
I20260620 23:08:13.498253 26660 tablet_replica.cc:330] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: stopping tablet replica
I20260620 23:08:13.498703 26660 raft_consensus.cc:2229] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:08:13.499233 26660 raft_consensus.cc:2258] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:13.499192 26662 raft_consensus.cc:2229] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:08:13.499977 26661 tablet_replica.cc:330] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: stopping tablet replica
I20260620 23:08:13.500182 26662 raft_consensus.cc:2258] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:13.500358 26661 raft_consensus.cc:2229] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:08:13.500963 26661 raft_consensus.cc:2258] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:13.502231 26662 ts_tablet_manager.cc:1905] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:13.502408 26660 ts_tablet_manager.cc:1905] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:13.506994 26660 ts_tablet_manager.cc:1918] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:13.507162 26660 log.cc:1196] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/5ff852fb3a0b4d5cb1aa1d634317e10f
I20260620 23:08:13.507660 26660 ts_tablet_manager.cc:1939] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Deleting consensus metadata
I20260620 23:08:13.508033 26661 ts_tablet_manager.cc:1905] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:13.509369 26150 catalog_manager.cc:4841] TS 83c3f45d312d41ee98c8c5bf4dbfb7ee (127.12.2.66:38259): tablet 5ff852fb3a0b4d5cb1aa1d634317e10f (table test-table [id=7445a3d5fde04340a3d15ff15581278c]) successfully deleted
I20260620 23:08:13.512356 26662 ts_tablet_manager.cc:1918] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:13.512519 26662 log.cc:1196] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/5ff852fb3a0b4d5cb1aa1d634317e10f
I20260620 23:08:13.513028 26662 ts_tablet_manager.cc:1939] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 1b484a0c2cf44b0a8e3f52ef9291a968: Deleting consensus metadata
I20260620 23:08:13.514693 26148 catalog_manager.cc:4841] TS 1b484a0c2cf44b0a8e3f52ef9291a968 (127.12.2.65:36499): tablet 5ff852fb3a0b4d5cb1aa1d634317e10f (table test-table [id=7445a3d5fde04340a3d15ff15581278c]) successfully deleted
I20260620 23:08:13.515401 26661 ts_tablet_manager.cc:1918] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:13.515563 26661 log.cc:1196] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/5ff852fb3a0b4d5cb1aa1d634317e10f
I20260620 23:08:13.516063 26661 ts_tablet_manager.cc:1939] T 5ff852fb3a0b4d5cb1aa1d634317e10f P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Deleting consensus metadata
I20260620 23:08:13.517444 26149 catalog_manager.cc:4841] TS 6ee80df0b37c4d8aa6f4e70d9ee0943a (127.12.2.67:34451): tablet 5ff852fb3a0b4d5cb1aa1d634317e10f (table test-table [id=7445a3d5fde04340a3d15ff15581278c]) successfully deleted
Jun 20 23:08:13 dist-test-slave-7g8g krb5kdc[26119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:08:13.525712 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.489262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.12.2.65:36499)
0620 23:08:13.489429 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:13.489435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:13.509355 (+ 19920us) server_negotiation.cc:407] Connection header received
0620 23:08:13.509823 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:13.509829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:13.509917 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:13.509996 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:13.511907 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.513034 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.515099 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.515711 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.515840 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.520549 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:13.520560 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:13.520574 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:13.520605 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:13.523178 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:13.523848 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:13.523855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:13.523857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:13.523975 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:13.524321 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:13.524324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:13.524327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:13.524652 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:13.524987 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:13.525199 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 23:08:13.525402 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:08:13.526314 26348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:13.614457 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 26225
I20260620 23:08:13.635668 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 26359
Jun 20 23:08:13 dist-test-slave-7g8g krb5kdc[26119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:08:13.653062 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.636454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.12.2.66:38259)
0620 23:08:13.636646 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:13.636651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:13.636776 (+ 125us) server_negotiation.cc:407] Connection header received
0620 23:08:13.636966 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:13.636972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:13.637079 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:13.637175 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:13.638797 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.639641 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.641496 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.641869 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.641956 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.646759 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:13.646776 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:13.646796 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:13.646843 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:13.650367 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:13.651022 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:13.651032 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:13.651036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:13.651185 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:13.651520 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:13.651525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:13.651528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:13.651978 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:13.652254 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:13.652485 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 23:08:13.652707 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:08:13.653759 26482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:13.742393 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 26359
I20260620 23:08:13.762620 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 26492
Jun 20 23:08:13 dist-test-slave-7g8g krb5kdc[26119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:08:13.778911 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:13.763354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.12.2.67:34451)
0620 23:08:13.763533 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:13.763537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:13.763694 (+ 157us) server_negotiation.cc:407] Connection header received
0620 23:08:13.763845 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:13.763850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:13.763934 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:13.764055 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:13.765340 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.766534 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.767894 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:13.768532 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:13.768691 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:13.773007 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:13.773023 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:13.773038 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:13.773070 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:13.776099 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:13.776926 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:13.776937 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:13.776939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:13.777079 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:13.777494 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:13.777497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:13.777498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:13.777866 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:13.778095 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:13.778420 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:08:13.778573 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:08:13.779613 26615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:13.899012 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 26492
I20260620 23:08:13.914242 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 26132
I20260620 23:08:13.915282 26194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:14.024410 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 26132
I20260620 23:08:14.041580 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:36055
--webserver_interface=127.12.2.126
--webserver_port=41835
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:36055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:14.353397 26668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:14.353683 26668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:14.353814 26668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:14.361006 26668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:08:14.361109 26668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:14.361177 26668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:14.361254 26668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:08:14.361308 26668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:08:14.371390 26668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:36055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:36055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=41835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:14.372658 26668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:14.373924 26668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:14.394245 26674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:14.394284 26675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:14.394213 26677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:14.395211 26668 server_base.cc:1034] running on GCE node
I20260620 23:08:14.396111 26668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:14.397693 26668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:14.398895 26668 hybrid_clock.cc:648] HybridClock initialized: now 1781996894398842 us; error 45 us; skew 500 ppm
I20260620 23:08:14.399287 26668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:14 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996894, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:14.402603 26668 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:08:14.404230 26668 webserver.cc:460] Webserver started at http://127.12.2.126:41835/ using document root <none> and password file <none>
I20260620 23:08:14.404920 26668 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:14.405048 26668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:14.410878 26668 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:08:14.413625 26684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:14.414927 26668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:08:14.415113 26668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "b26c4faf5ad84fcf901967b09261e0e9"
format_stamp: "Formatted at 2026-06-20 23:08:11 on dist-test-slave-7g8g"
server_key: "8616e5425536cbf0897fc64f906cc396"
server_key_iv: "bcca3427a183a5f31e9bfeed473c3594"
server_key_version: "encryptionkey@0"
I20260620 23:08:14.416136 26668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:14.432940 26668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:14.437043 26668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:14.437310 26668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:14.454787 26668 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:36055
I20260620 23:08:14.454808 26735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:36055 every 8 connection(s)
I20260620 23:08:14.456243 26668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:08:14.461688 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 26668
I20260620 23:08:14.462615 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:36499
--local_ip_for_outbound_sockets=127.12.2.65
--tserver_master_addrs=127.12.2.126:36055
--webserver_port=43467
--webserver_interface=127.12.2.65
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:08:14.463543 26736 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:08:14.467988 26736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Bootstrap starting.
I20260620 23:08:14.487587 26736 log.cc:824] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:14.499365 26736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:08:14.499861 26736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Bootstrap complete.
I20260620 23:08:14.508638 26736 raft_consensus.cc:348] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:14.509471 26736 raft_consensus.cc:729] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26c4faf5ad84fcf901967b09261e0e9, State: Initialized, Role: FOLLOWER
I20260620 23:08:14.510031 26736 consensus_queue.cc:260] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:14.510252 26736 raft_consensus.cc:388] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:08:14.510354 26736 raft_consensus.cc:482] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:08:14.510500 26736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:08:14.514163 26736 raft_consensus.cc:504] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:14.514638 26736 leader_election.cc:302] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b26c4faf5ad84fcf901967b09261e0e9; no voters:
I20260620 23:08:14.515249 26736 leader_election.cc:288] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:08:14.515383 26740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:08:14.516396 26740 raft_consensus.cc:686] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 2 LEADER]: Becoming Leader. State: Replica: b26c4faf5ad84fcf901967b09261e0e9, State: Running, Role: LEADER
I20260620 23:08:14.517009 26740 consensus_queue.cc:237] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:14.517422 26736 sys_catalog.cc:564] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:08:14.519384 26741 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b26c4faf5ad84fcf901967b09261e0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } } }
I20260620 23:08:14.519775 26741 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:14.520561 26742 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b26c4faf5ad84fcf901967b09261e0e9. Latest consensus state: current_term: 2 leader_uuid: "b26c4faf5ad84fcf901967b09261e0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } } }
I20260620 23:08:14.520830 26742 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:14.528023 26745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:08:14.536540 26745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:08:14.538053 26745 catalog_manager.cc:1259] Loaded cluster ID: feffd4ee8df845c1aeb8e058885901aa
I20260620 23:08:14.538220 26745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:08:14.541464 26745 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 23:08:14.543562 26757 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:08:14.557823 26745 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 23:08:14.567054 26745 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:08:14.568827 26745 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Loaded TSK: 0
I20260620 23:08:14.570356 26745 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:08:14.799544 26738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:14.799790 26738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:14.799907 26738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:14.807032 26738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:14.807132 26738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:14.807287 26738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:08:14.816891 26738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:36499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=43467
--tserver_master_addrs=127.12.2.126:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:14.818147 26738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:14.819520 26738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:14.834282 26764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:14.834302 26763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:14.835072 26738 server_base.cc:1034] running on GCE node
W20260620 23:08:14.835196 26766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:14.835834 26738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:14.836735 26738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:14.837944 26738 hybrid_clock.cc:648] HybridClock initialized: now 1781996894837880 us; error 52 us; skew 500 ppm
I20260620 23:08:14.838246 26738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:14 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996894, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:14.841310 26738 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:08:14.842733 26738 webserver.cc:460] Webserver started at http://127.12.2.65:43467/ using document root <none> and password file <none>
I20260620 23:08:14.843406 26738 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:14.843534 26738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:14.848783 26738 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:08:14.851478 26773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:14.852825 26738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:08:14.852967 26738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "1b484a0c2cf44b0a8e3f52ef9291a968"
format_stamp: "Formatted at 2026-06-20 23:08:12 on dist-test-slave-7g8g"
server_key: "12af4a1da52b8fe7228c2eb5878a5b59"
server_key_iv: "e0fffb4b1adf40a818b433968afab846"
server_key_version: "encryptionkey@0"
I20260620 23:08:14.853884 26738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:14.869416 26738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:14.873389 26738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:14.873636 26738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:14.874657 26738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:14.876066 26738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:14.876159 26738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:14.876263 26738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:14.876334 26738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:14.906292 26738 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:36499
I20260620 23:08:14.906366 26885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:36499 every 8 connection(s)
I20260620 23:08:14.907738 26738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:08:14.916499 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 26738
I20260620 23:08:14.917049 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:38259
--local_ip_for_outbound_sockets=127.12.2.66
--tserver_master_addrs=127.12.2.126:36055
--webserver_port=33509
--webserver_interface=127.12.2.66
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:08:14 dist-test-slave-7g8g krb5kdc[26119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996894, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:14.929876 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:14.910692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:48647)
0620 23:08:14.911481 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:14.911519 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:14.913164 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:14.914437 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:14.914447 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:14.915053 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:14.915568 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:14.915584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:14.917306 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:14.917311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:14.918609 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:14.918621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:14.919773 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:14.919779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:14.920083 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:14.920782 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:14.920804 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:14.922794 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:14.925827 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:14.925835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:14.925850 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:14.926239 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:14.926634 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:14.926639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:14.926643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:14.926817 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:14.927530 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:14.927537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:14.927849 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 23:08:14.928082 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 23:08:14.928603 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":323,"threads_started":1}
I20260620 23:08:14.929876 26888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:14.910515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:48647 (local address 127.12.2.126:36055)
0620 23:08:14.911220 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:14.911237 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:14.911635 (+ 398us) server_negotiation.cc:407] Connection header received
0620 23:08:14.913651 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:14.913663 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:14.913972 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:14.914282 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:14.915770 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:14.917104 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:14.918734 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:14.919526 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:14.919675 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:14.923056 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:14.923064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:14.923077 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:14.923111 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:14.925694 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:14.926360 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:14.926366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:14.926371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:14.926531 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:14.926917 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:14.926920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:14.926922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:14.927400 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:14.927582 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:14.928293 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 23:08:14.928655 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":297,"threads_started":1}
I20260620 23:08:14.933034 26886 heartbeater.cc:347] Connected to a master server at 127.12.2.126:36055
I20260620 23:08:14.933303 26886 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:14.934019 26886 heartbeater.cc:511] Master 127.12.2.126:36055 requested a full tablet report, sending...
I20260620 23:08:14.936388 26701 ts_manager.cc:194] Registered new tserver with Master: 1b484a0c2cf44b0a8e3f52ef9291a968 (127.12.2.65:36499)
I20260620 23:08:14.938541 26701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:48647
W20260620 23:08:15.224951 26890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:15.225212 26890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:15.225301 26890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:15.232458 26890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:15.232568 26890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:15.232679 26890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:08:15.242106 26890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:38259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=33509
--tserver_master_addrs=127.12.2.126:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:15.243302 26890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:15.244534 26890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:15.256560 26896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:15.256560 26895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:15.256641 26898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:15.257378 26890 server_base.cc:1034] running on GCE node
I20260620 23:08:15.257930 26890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:15.258798 26890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:15.259987 26890 hybrid_clock.cc:648] HybridClock initialized: now 1781996895259957 us; error 46 us; skew 500 ppm
I20260620 23:08:15.260272 26890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:15 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996895, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:15.263183 26890 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:08:15.264484 26890 webserver.cc:460] Webserver started at http://127.12.2.66:33509/ using document root <none> and password file <none>
I20260620 23:08:15.265125 26890 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:15.265239 26890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:15.270191 26890 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:08:15.272850 26905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:15.274185 26890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:08:15.274329 26890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee"
format_stamp: "Formatted at 2026-06-20 23:08:12 on dist-test-slave-7g8g"
server_key: "2688aad9ef9ac6795b37a6f8937aad92"
server_key_iv: "3760f7740b9ee20dcd56a682e4247bef"
server_key_version: "encryptionkey@0"
I20260620 23:08:15.275314 26890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:15.292321 26890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:15.296402 26890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:15.296700 26890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:15.297899 26890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:15.299583 26890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:15.299700 26890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:15.299805 26890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:15.299875 26890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:15.329530 26890 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:38259
I20260620 23:08:15.329595 27017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:38259 every 8 connection(s)
I20260620 23:08:15.330946 26890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:08:15.337931 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 26890
I20260620 23:08:15.338543 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:34451
--local_ip_for_outbound_sockets=127.12.2.67
--tserver_master_addrs=127.12.2.126:36055
--webserver_port=38433
--webserver_interface=127.12.2.67
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:08:15 dist-test-slave-7g8g krb5kdc[26119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996895, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:15.351130 26888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.333364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:52839 (local address 127.12.2.126:36055)
0620 23:08:15.333538 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:15.333542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:15.334666 (+ 1124us) server_negotiation.cc:407] Connection header received
0620 23:08:15.336359 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:15.336363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:15.336430 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:15.336523 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:15.338084 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.338836 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.340966 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.341396 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.341541 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.345573 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:15.345578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:15.345582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:15.345616 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:15.348080 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:15.348875 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:15.348879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:15.348881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:15.349016 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:15.349476 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:15.349480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:15.349482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:15.349743 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:15.349855 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:15.350607 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 23:08:15.350820 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:08:15.352377 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.333835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:52839)
0620 23:08:15.334490 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:15.334532 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:15.336113 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:15.336730 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:15.336739 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:15.337215 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:15.337890 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.337917 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.339023 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.339030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.340817 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.340831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.341662 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.341669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.341951 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.342633 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:15.342660 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:15.345331 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:15.348215 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:15.348225 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:15.348242 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:15.348760 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:15.349108 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:15.349113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:15.349116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:15.349337 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:15.349858 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:15.349865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:15.350260 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 23:08:15.350535 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 23:08:15.351050 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":255,"threads_started":1}
I20260620 23:08:15.353754 27018 heartbeater.cc:347] Connected to a master server at 127.12.2.126:36055
I20260620 23:08:15.354038 27018 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:15.354741 27018 heartbeater.cc:511] Master 127.12.2.126:36055 requested a full tablet report, sending...
I20260620 23:08:15.356145 26701 ts_manager.cc:194] Registered new tserver with Master: 83c3f45d312d41ee98c8c5bf4dbfb7ee (127.12.2.66:38259)
I20260620 23:08:15.356931 26701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:52839
W20260620 23:08:15.657853 27021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:15.658170 27021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:15.658308 27021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:15.665495 27021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:15.665618 27021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:15.665766 27021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:08:15.675320 27021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:34451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=38433
--tserver_master_addrs=127.12.2.126:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:15.676604 27021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:15.677894 27021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:15.690090 27029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:15.690176 27026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:15.690153 27027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:15.691236 27021 server_base.cc:1034] running on GCE node
I20260620 23:08:15.691829 27021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:15.692830 27021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:15.694042 27021 hybrid_clock.cc:648] HybridClock initialized: now 1781996895693990 us; error 49 us; skew 500 ppm
I20260620 23:08:15.694329 27021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:15 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996895, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:15.697561 27021 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:08:15.698953 27021 webserver.cc:460] Webserver started at http://127.12.2.67:38433/ using document root <none> and password file <none>
I20260620 23:08:15.699630 27021 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:15.699739 27021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:15.705056 27021 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 23:08:15.707717 27036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:15.709189 27021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:08:15.709352 27021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a"
format_stamp: "Formatted at 2026-06-20 23:08:13 on dist-test-slave-7g8g"
server_key: "b97be12e711a60e28eab4d6b6e6c3ee7"
server_key_iv: "4df8e949d9170130e0535e3bcde72cb5"
server_key_version: "encryptionkey@0"
I20260620 23:08:15.710481 27021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:15.735311 27021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:15.739508 27021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:15.739733 27021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:15.740667 27021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:15.741963 27021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:15.742060 27021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:15.742139 27021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:15.742192 27021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:15.771363 27021 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:34451
I20260620 23:08:15.771430 27148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:34451 every 8 connection(s)
I20260620 23:08:15.772814 27021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:08:15.782224 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 27021
Jun 20 23:08:15 dist-test-slave-7g8g krb5kdc[26119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996895, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:15.792671 26888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.775221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:58139 (local address 127.12.2.126:36055)
0620 23:08:15.775418 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:15.775424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:15.776685 (+ 1261us) server_negotiation.cc:407] Connection header received
0620 23:08:15.778403 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:15.778408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:15.778477 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:15.778578 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:15.780065 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.780794 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.782399 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.782832 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.782977 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.786997 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:15.787004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:15.787008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:15.787042 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:15.789575 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:15.790303 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:15.790309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:15.790313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:15.790443 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:15.790851 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:15.790856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:15.790860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:15.791182 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:15.791334 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:15.792176 (+ 842us) server_negotiation.cc:299] Negotiation successful
0620 23:08:15.792407 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:08:15.794037 27151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.775731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:58139)
0620 23:08:15.776473 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:15.776527 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:15.778154 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:15.778758 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:15.778766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:15.779345 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:15.779907 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.779924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.780952 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.780957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.782230 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.782242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.783096 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.783104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.783491 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.784188 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:15.784218 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:15.786750 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:15.789743 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:15.789750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:15.789764 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:15.790180 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:15.790550 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:15.790555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:15.790557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:15.790740 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:15.791298 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:15.791306 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:15.791756 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 23:08:15.792060 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 23:08:15.792582 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":287,"threads_started":1}
I20260620 23:08:15.795572 27149 heartbeater.cc:347] Connected to a master server at 127.12.2.126:36055
I20260620 23:08:15.795928 27149 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:15.796934 27149 heartbeater.cc:511] Master 127.12.2.126:36055 requested a full tablet report, sending...
I20260620 23:08:15.797513 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.783144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.12.2.126:36055)
0620 23:08:15.784063 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:15.784067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:15.784203 (+ 136us) server_negotiation.cc:407] Connection header received
0620 23:08:15.784421 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:15.784425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:15.784486 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:15.784582 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:15.786287 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.787071 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.788433 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.788881 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.788984 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.791479 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:15.791485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:15.791489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:15.791522 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:15.794702 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:15.795405 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:15.795410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:15.795413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:15.795550 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:15.796163 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:15.796167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:15.796169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:15.796427 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:15.796638 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:15.797055 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 23:08:15.797253 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":711,"threads_started":1}
I20260620 23:08:15.798514 26701 ts_manager.cc:194] Registered new tserver with Master: 6ee80df0b37c4d8aa6f4e70d9ee0943a (127.12.2.67:34451)
I20260620 23:08:15.799408 26701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:58139
I20260620 23:08:15.800082 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:08:15.806661 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.801143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.12.2.126:36055)
0620 23:08:15.801318 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:15.801324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:15.801723 (+ 399us) server_negotiation.cc:407] Connection header received
0620 23:08:15.801894 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:15.801899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:15.802062 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:15.802197 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:08:15.802514 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.803245 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.804668 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.805095 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.805185 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.805694 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:08:15.806187 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:08:15.806295 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:15.806392 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 23:08:15.806458 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:08:15.807628 26701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:08:15.814823 26701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:08:15.836889 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.826391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:38259 (local address 127.0.0.1:38676)
0620 23:08:15.827361 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:15.827396 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:15.827556 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:15.829825 (+ 2269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:15.829834 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:15.829874 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:15.830305 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.830322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.832478 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.832483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.834429 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.834454 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.835964 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.835971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.836164 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.836226 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 23:08:15.836329 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 23:08:15.836527 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":618,"threads_started":1}
I20260620 23:08:15.837530 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.826478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.12.2.66:38259)
0620 23:08:15.829135 (+ 2657us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:15.829146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:15.829182 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:08:15.829285 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:15.829294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:15.829588 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:15.830542 (+ 954us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:15.830812 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.832317 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.834586 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.835737 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.836452 (+ 715us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.836680 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:15.836747 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 23:08:15.836909 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2476,"thread_start_us":2304,"threads_started":1}
I20260620 23:08:15.838155 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.826494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:36499 (local address 127.0.0.1:36262)
0620 23:08:15.828113 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:15.828130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:15.828620 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:15.830573 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:15.830578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:15.830605 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:15.830990 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.831001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.833483 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.833487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.835614 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.835626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.837476 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.837483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.837675 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.837704 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:08:15.837787 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:08:15.837902 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1492,"mutex_wait_us":1,"thread_start_us":1362,"threads_started":1}
I20260620 23:08:15.838847 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.827661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.12.2.65:36499)
0620 23:08:15.829738 (+ 2077us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:15.829746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:15.829772 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:08:15.829878 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:15.829886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:15.830123 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:15.830321 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:15.831507 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.833345 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.835791 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.837220 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.837378 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.837562 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:15.838430 (+ 868us) server_negotiation.cc:299] Negotiation successful
0620 23:08:15.838564 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1925,"thread_start_us":1938,"threads_started":1}
I20260620 23:08:15.840796 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.826862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:34451 (local address 127.0.0.1:39512)
0620 23:08:15.831976 (+ 5114us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:15.831991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:15.832099 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:15.833103 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:15.833108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:15.833136 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:15.833512 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.833524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.835468 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.835473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.837539 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.837553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.840223 (+ 2670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.840229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.840388 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.840417 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:08:15.840496 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:08:15.840591 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4999,"thread_start_us":4869,"threads_started":1}
I20260620 23:08:15.841388 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.827229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.12.2.67:34451)
0620 23:08:15.831561 (+ 4332us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:15.831570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:15.832089 (+ 519us) server_negotiation.cc:407] Connection header received
0620 23:08:15.832372 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:15.832384 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:15.832646 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:15.832824 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:15.833735 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.835321 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.837690 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.839740 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.839853 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.840080 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:15.841042 (+ 962us) server_negotiation.cc:299] Negotiation successful
0620 23:08:15.841164 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4171,"thread_start_us":4003,"threads_started":1}
I20260620 23:08:15.841559 26953 tablet_service.cc:1467] Processing CreateTablet for tablet 11eb35eab6f845bbb5c190532129fbc9 (DEFAULT_TABLE table=test-table [id=a6ba46046b1440f6ac081e1fe714e10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:15.842149 26821 tablet_service.cc:1467] Processing CreateTablet for tablet 11eb35eab6f845bbb5c190532129fbc9 (DEFAULT_TABLE table=test-table [id=a6ba46046b1440f6ac081e1fe714e10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:15.843048 26953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11eb35eab6f845bbb5c190532129fbc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:15.843534 26821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11eb35eab6f845bbb5c190532129fbc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:15.844053 27084 tablet_service.cc:1467] Processing CreateTablet for tablet 11eb35eab6f845bbb5c190532129fbc9 (DEFAULT_TABLE table=test-table [id=a6ba46046b1440f6ac081e1fe714e10e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:15.845101 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11eb35eab6f845bbb5c190532129fbc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:15.856799 27165 tablet_bootstrap.cc:492] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Bootstrap starting.
I20260620 23:08:15.859617 27167 tablet_bootstrap.cc:492] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Bootstrap starting.
I20260620 23:08:15.860795 27165 tablet_bootstrap.cc:654] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:15.862169 27165 log.cc:824] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:15.863143 27167 tablet_bootstrap.cc:654] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:15.863566 27166 tablet_bootstrap.cc:492] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: Bootstrap starting.
I20260620 23:08:15.864351 27167 log.cc:824] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:15.865559 27165 tablet_bootstrap.cc:492] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a: No bootstrap required, opened a new log
I20260620 23:08:15.865803 27165 ts_tablet_manager.cc:1397] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:08:15.866732 27166 tablet_bootstrap.cc:654] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:15.867305 27167 tablet_bootstrap.cc:492] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee: No bootstrap required, opened a new log
I20260620 23:08:15.867573 27167 ts_tablet_manager.cc:1397] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 23:08:15.867941 27166 log.cc:824] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:15.871208 27166 tablet_bootstrap.cc:492] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: No bootstrap required, opened a new log
I20260620 23:08:15.871466 27166 ts_tablet_manager.cc:1397] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260620 23:08:15.873337 27165 raft_consensus.cc:348] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } }
I20260620 23:08:15.873692 27165 raft_consensus.cc:374] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:15.873781 27165 raft_consensus.cc:729] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee80df0b37c4d8aa6f4e70d9ee0943a, State: Initialized, Role: FOLLOWER
I20260620 23:08:15.874341 27165 consensus_queue.cc:260] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } }
I20260620 23:08:15.875396 27149 heartbeater.cc:503] Master 127.12.2.126:36055 was elected leader, sending a full tablet report...
I20260620 23:08:15.876078 27165 ts_tablet_manager.cc:1428] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 23:08:15.877300 27167 raft_consensus.cc:348] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } }
I20260620 23:08:15.877732 27167 raft_consensus.cc:374] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:15.877827 27167 raft_consensus.cc:729] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c3f45d312d41ee98c8c5bf4dbfb7ee, State: Initialized, Role: FOLLOWER
I20260620 23:08:15.878412 27167 consensus_queue.cc:260] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } }
I20260620 23:08:15.881754 27166 raft_consensus.cc:348] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } }
I20260620 23:08:15.882205 27166 raft_consensus.cc:374] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:15.882302 27166 raft_consensus.cc:729] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b484a0c2cf44b0a8e3f52ef9291a968, State: Initialized, Role: FOLLOWER
I20260620 23:08:15.882903 27166 consensus_queue.cc:260] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } }
I20260620 23:08:15.883971 26886 heartbeater.cc:503] Master 127.12.2.126:36055 was elected leader, sending a full tablet report...
I20260620 23:08:15.884698 27166 ts_tablet_manager.cc:1428] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 23:08:15.885490 27018 heartbeater.cc:503] Master 127.12.2.126:36055 was elected leader, sending a full tablet report...
I20260620 23:08:15.886077 27167 ts_tablet_manager.cc:1428] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260620 23:08:15.909807 26887 tablet.cc:2369] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:08:15.934837 27172 raft_consensus.cc:482] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:08:15.935180 27172 raft_consensus.cc:504] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } }
I20260620 23:08:15.936728 27172 leader_election.cc:288] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b484a0c2cf44b0a8e3f52ef9291a968 (127.12.2.65:36499), 6ee80df0b37c4d8aa6f4e70d9ee0943a (127.12.2.67:34451)
I20260620 23:08:15.945225 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.937142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:50881 (local address 127.12.2.67:34451)
0620 23:08:15.937347 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:15.937353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:15.937993 (+ 640us) server_negotiation.cc:407] Connection header received
0620 23:08:15.938137 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:15.938143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:15.938214 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:15.938294 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:15.939074 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.940169 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.942897 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.944122 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.944231 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.944330 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:15.944840 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 23:08:15.944966 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:08:15.945225 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.937111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:56701 (local address 127.12.2.65:36499)
0620 23:08:15.937488 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:15.937494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:15.937852 (+ 358us) server_negotiation.cc:407] Connection header received
0620 23:08:15.938082 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:15.938087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:15.938166 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:15.938259 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:15.939110 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.940160 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.942892 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.944096 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.944237 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.944328 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:15.944840 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 23:08:15.944966 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260620 23:08:15.945389 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.936976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:36499 (local address 127.12.2.66:56701)
0620 23:08:15.937741 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:15.937768 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:15.937925 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:15.938304 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:15.938309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:15.938338 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:15.938982 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.938994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.940279 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.940285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.942726 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.942738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.944385 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.944390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.944667 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.944718 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:08:15.944818 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:08:15.944972 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":188,"thread_start_us":384,"threads_started":1}
I20260620 23:08:15.945622 27175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.937079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:34451 (local address 127.12.2.66:50881)
0620 23:08:15.937899 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:15.937914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:15.938017 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:15.938310 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:15.938314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:15.938396 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:15.938847 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.938861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.940292 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.940296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.942724 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:15.942738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.944382 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.944390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:15.944664 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.944724 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 23:08:15.944817 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:08:15.944973 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":146,"spinlock_wait_cycles":22784,"thread_start_us":526,"threads_started":1}
I20260620 23:08:15.946410 26841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11eb35eab6f845bbb5c190532129fbc9" candidate_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" is_pre_election: true
I20260620 23:08:15.946575 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11eb35eab6f845bbb5c190532129fbc9" candidate_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" is_pre_election: true
I20260620 23:08:15.947070 26841 raft_consensus.cc:2454] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c3f45d312d41ee98c8c5bf4dbfb7ee in term 0.
I20260620 23:08:15.947086 27104 raft_consensus.cc:2454] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c3f45d312d41ee98c8c5bf4dbfb7ee in term 0.
I20260620 23:08:15.947767 26906 leader_election.cc:302] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b484a0c2cf44b0a8e3f52ef9291a968, 83c3f45d312d41ee98c8c5bf4dbfb7ee; no voters:
I20260620 23:08:15.948136 27172 raft_consensus.cc:2781] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:08:15.948272 27172 raft_consensus.cc:482] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:08:15.948369 27172 raft_consensus.cc:3037] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:15.950557 27172 raft_consensus.cc:504] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } }
I20260620 23:08:15.951192 27172 leader_election.cc:288] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [CANDIDATE]: Term 1 election: Requested vote from peers 1b484a0c2cf44b0a8e3f52ef9291a968 (127.12.2.65:36499), 6ee80df0b37c4d8aa6f4e70d9ee0943a (127.12.2.67:34451)
I20260620 23:08:15.951579 26841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11eb35eab6f845bbb5c190532129fbc9" candidate_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968"
I20260620 23:08:15.951665 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11eb35eab6f845bbb5c190532129fbc9" candidate_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a"
I20260620 23:08:15.951761 26841 raft_consensus.cc:3037] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:15.951859 27104 raft_consensus.cc:3037] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:15.954257 26841 raft_consensus.cc:2454] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c3f45d312d41ee98c8c5bf4dbfb7ee in term 1.
I20260620 23:08:15.954345 27104 raft_consensus.cc:2454] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c3f45d312d41ee98c8c5bf4dbfb7ee in term 1.
I20260620 23:08:15.954618 26906 leader_election.cc:302] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b484a0c2cf44b0a8e3f52ef9291a968, 83c3f45d312d41ee98c8c5bf4dbfb7ee; no voters:
I20260620 23:08:15.954891 27172 raft_consensus.cc:2781] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:08:15.955328 27172 raft_consensus.cc:686] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 1 LEADER]: Becoming Leader. State: Replica: 83c3f45d312d41ee98c8c5bf4dbfb7ee, State: Running, Role: LEADER
I20260620 23:08:15.955770 27172 consensus_queue.cc:237] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } } peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } }
I20260620 23:08:15.959959 26701 catalog_manager.cc:5495] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c3f45d312d41ee98c8c5bf4dbfb7ee (127.12.2.66). New cstate: current_term: 1 leader_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 38259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 } health_report { overall_health: UNKNOWN } } }
I20260620 23:08:15.996109 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:15.990636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.12.2.66:38259)
0620 23:08:15.990855 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:15.990860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:15.990878 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:08:15.991001 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:15.991007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:15.991109 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:15.991242 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:08:15.991620 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.992464 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.993950 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:15.994363 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:15.994467 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:15.994992 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:08:15.995509 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:08:15.995653 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:15.995768 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 23:08:15.995848 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:08:16.004305 27104 raft_consensus.cc:1261] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 1 FOLLOWER]: Refusing update from remote peer 83c3f45d312d41ee98c8c5bf4dbfb7ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:16.004349 26841 raft_consensus.cc:1261] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 1 FOLLOWER]: Refusing update from remote peer 83c3f45d312d41ee98c8c5bf4dbfb7ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:16.005368 27176 consensus_queue.cc:1035] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:16.005712 27172 consensus_queue.cc:1035] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b484a0c2cf44b0a8e3f52ef9291a968" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:16.017239 27179 mvcc.cc:205] Tried to move back new op lower bound from 7299059286023643136 to 7299059285838618624. Current Snapshot: MvccSnapshot[applied={T|T < 7299059286023643136}]
I20260620 23:08:16.036175 26701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:08:16.036468 26701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:08:16.040119 26701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Sending DeleteTablet for 3 replicas of tablet 11eb35eab6f845bbb5c190532129fbc9
I20260620 23:08:16.041334 26821 tablet_service.cc:1514] Processing DeleteTablet for tablet 11eb35eab6f845bbb5c190532129fbc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:16 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:36262
I20260620 23:08:16.041373 26953 tablet_service.cc:1514] Processing DeleteTablet for tablet 11eb35eab6f845bbb5c190532129fbc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:16 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:38676
I20260620 23:08:16.041497 27084 tablet_service.cc:1514] Processing DeleteTablet for tablet 11eb35eab6f845bbb5c190532129fbc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:16 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:39512
I20260620 23:08:16.042728 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 26738
I20260620 23:08:16.054772 27189 tablet_replica.cc:330] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: stopping tablet replica
I20260620 23:08:16.055491 27189 raft_consensus.cc:2229] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:08:16.056252 27189 raft_consensus.cc:2258] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:16.056921 27191 tablet_replica.cc:330] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a: stopping tablet replica
I20260620 23:08:16.057368 27190 tablet_replica.cc:330] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee: stopping tablet replica
I20260620 23:08:16.058015 27191 raft_consensus.cc:2229] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:08:16.058527 27191 raft_consensus.cc:2258] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:16.059125 27189 ts_tablet_manager.cc:1905] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:16.060060 27190 raft_consensus.cc:2229] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:08:16.060628 27191 ts_tablet_manager.cc:1905] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:16.060837 27190 raft_consensus.cc:2258] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:16.063046 27190 ts_tablet_manager.cc:1905] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:16.065958 27189 ts_tablet_manager.cc:1918] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:16.066125 27189 log.cc:1196] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/11eb35eab6f845bbb5c190532129fbc9
I20260620 23:08:16.066646 27189 ts_tablet_manager.cc:1939] T 11eb35eab6f845bbb5c190532129fbc9 P 1b484a0c2cf44b0a8e3f52ef9291a968: Deleting consensus metadata
I20260620 23:08:16.067976 27191 ts_tablet_manager.cc:1918] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:16.068203 27191 log.cc:1196] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/11eb35eab6f845bbb5c190532129fbc9
I20260620 23:08:16.068610 26685 catalog_manager.cc:4841] TS 1b484a0c2cf44b0a8e3f52ef9291a968 (127.12.2.65:36499): tablet 11eb35eab6f845bbb5c190532129fbc9 (table test-table [id=a6ba46046b1440f6ac081e1fe714e10e]) successfully deleted
I20260620 23:08:16.068882 27191 ts_tablet_manager.cc:1939] T 11eb35eab6f845bbb5c190532129fbc9 P 6ee80df0b37c4d8aa6f4e70d9ee0943a: Deleting consensus metadata
I20260620 23:08:16.069243 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:16.043891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.12.2.65:36499)
0620 23:08:16.044166 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:16.044173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:16.044190 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:08:16.044249 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:16.044254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:16.044370 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:16.044473 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:16.046474 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:16.047732 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:16.049723 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:16.050412 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:16.050537 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:16.054408 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:16.054434 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:16.054471 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:16.054526 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:16.058589 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:16.059636 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:16.059645 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:16.059648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:16.059816 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:16.067457 (+ 7641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:16.067463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:16.067466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:16.068022 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:16.068337 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:16.068700 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 23:08:16.068917 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 23:08:16.069895 26881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:16.071048 26686 catalog_manager.cc:4841] TS 6ee80df0b37c4d8aa6f4e70d9ee0943a (127.12.2.67:34451): tablet 11eb35eab6f845bbb5c190532129fbc9 (table test-table [id=a6ba46046b1440f6ac081e1fe714e10e]) successfully deleted
I20260620 23:08:16.072173 27190 ts_tablet_manager.cc:1918] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:16.072374 27190 log.cc:1196] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/11eb35eab6f845bbb5c190532129fbc9
I20260620 23:08:16.072924 27190 ts_tablet_manager.cc:1939] T 11eb35eab6f845bbb5c190532129fbc9 P 83c3f45d312d41ee98c8c5bf4dbfb7ee: Deleting consensus metadata
I20260620 23:08:16.074847 26687 catalog_manager.cc:4841] TS 83c3f45d312d41ee98c8c5bf4dbfb7ee (127.12.2.66:38259): tablet 11eb35eab6f845bbb5c190532129fbc9 (table test-table [id=a6ba46046b1440f6ac081e1fe714e10e]) successfully deleted
I20260620 23:08:16.162555 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 26738
I20260620 23:08:16.176331 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 26890
I20260620 23:08:16.190011 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:16.177134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.12.2.66:38259)
0620 23:08:16.177329 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:16.177335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:16.177456 (+ 121us) server_negotiation.cc:407] Connection header received
0620 23:08:16.177650 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:16.177654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:16.177725 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:16.177812 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:16.179188 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:16.180048 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:16.181405 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:16.181856 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:16.181972 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:16.184496 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:16.184511 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:16.184531 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:16.184570 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:16.187450 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:16.188059 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:16.188066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:16.188068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:16.188195 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:16.188551 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:16.188555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:16.188557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:16.188905 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:16.189207 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:16.189480 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 23:08:16.189681 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:08:16.190624 27013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:16.301707 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 26890
I20260620 23:08:16.316453 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 27021
I20260620 23:08:16.329080 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:16.317116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.12.2.67:34451)
0620 23:08:16.317306 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:16.317310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:16.317332 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:08:16.317581 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:16.317585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:16.317652 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:16.317751 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:16.319078 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:16.319865 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:16.321188 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:16.321592 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:16.321688 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:16.324191 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:16.324204 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:16.324220 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:16.324253 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:16.326666 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:16.327299 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:16.327306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:16.327308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:16.327441 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:16.327795 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:16.327798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:16.327799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:16.328140 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:16.328367 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:16.328617 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 23:08:16.328774 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:08:16.329655 27144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:16.415704 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 27021
I20260620 23:08:16.428675 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 26668
I20260620 23:08:16.429821 26731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:16.539166 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 26668
I20260620 23:08:16.556172 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:36055
--webserver_interface=127.12.2.126
--webserver_port=41835
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:36055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:16.864758 27196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:16.865041 27196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:16.865188 27196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:16.872195 27196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:08:16.872316 27196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:16.872398 27196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:16.872473 27196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:08:16.872531 27196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:08:16.882025 27196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:36055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:36055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=41835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:16.883322 27196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:16.884553 27196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:16.895330 27202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:16.895340 27204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:16.895886 27201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:16.896159 27196 server_base.cc:1034] running on GCE node
I20260620 23:08:16.896878 27196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:16.898156 27196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:16.899345 27196 hybrid_clock.cc:648] HybridClock initialized: now 1781996896899290 us; error 41 us; skew 500 ppm
I20260620 23:08:16.899646 27196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:16 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996896, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:16.907390 27196 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:08:16.908831 27196 webserver.cc:460] Webserver started at http://127.12.2.126:41835/ using document root <none> and password file <none>
I20260620 23:08:16.909472 27196 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:16.909595 27196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:16.914765 27196 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:08:16.917349 27211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:16.918563 27196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:08:16.918707 27196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "b26c4faf5ad84fcf901967b09261e0e9"
format_stamp: "Formatted at 2026-06-20 23:08:11 on dist-test-slave-7g8g"
server_key: "8616e5425536cbf0897fc64f906cc396"
server_key_iv: "bcca3427a183a5f31e9bfeed473c3594"
server_key_version: "encryptionkey@0"
I20260620 23:08:16.919633 27196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:16.940263 27196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:16.944119 27196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:16.944365 27196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:16.961138 27196 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:36055
I20260620 23:08:16.961174 27262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:36055 every 8 connection(s)
I20260620 23:08:16.962538 27196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:08:16.965939 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 27196
I20260620 23:08:16.966925 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:36499
--local_ip_for_outbound_sockets=127.12.2.65
--tserver_master_addrs=127.12.2.126:36055
--webserver_port=43467
--webserver_interface=127.12.2.65
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:08:16.970367 27263 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:08:16.974777 27263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Bootstrap starting.
I20260620 23:08:16.998077 27263 log.cc:824] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:17.019327 27263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:08:17.019971 27263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9: Bootstrap complete.
I20260620 23:08:17.033305 27263 raft_consensus.cc:348] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:17.034261 27263 raft_consensus.cc:729] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26c4faf5ad84fcf901967b09261e0e9, State: Initialized, Role: FOLLOWER
I20260620 23:08:17.034838 27263 consensus_queue.cc:260] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:17.035084 27263 raft_consensus.cc:388] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:08:17.035198 27263 raft_consensus.cc:482] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:08:17.035351 27263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:08:17.038882 27263 raft_consensus.cc:504] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:17.039376 27263 leader_election.cc:302] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b26c4faf5ad84fcf901967b09261e0e9; no voters:
I20260620 23:08:17.039966 27263 leader_election.cc:288] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:08:17.040139 27267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:08:17.041128 27267 raft_consensus.cc:686] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [term 3 LEADER]: Becoming Leader. State: Replica: b26c4faf5ad84fcf901967b09261e0e9, State: Running, Role: LEADER
I20260620 23:08:17.041676 27267 consensus_queue.cc:237] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } }
I20260620 23:08:17.041978 27263 sys_catalog.cc:564] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:08:17.044373 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b26c4faf5ad84fcf901967b09261e0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } } }
I20260620 23:08:17.045138 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:17.044523 27269 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b26c4faf5ad84fcf901967b09261e0e9. Latest consensus state: current_term: 3 leader_uuid: "b26c4faf5ad84fcf901967b09261e0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b26c4faf5ad84fcf901967b09261e0e9" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36055 } } }
I20260620 23:08:17.045715 27269 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26c4faf5ad84fcf901967b09261e0e9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:17.056069 27276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:08:17.064268 27276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:08:17.068048 27276 catalog_manager.cc:1259] Loaded cluster ID: feffd4ee8df845c1aeb8e058885901aa
I20260620 23:08:17.068135 27276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 23:08:17.069587 27276 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa6b9f389ad google::LogMessage::Fail() at ??:0
@ 0x7fa6b9f3b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa6b9f384de google::LogMessage::Flush() at ??:0
@ 0x7fa6b9f39a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa6cf96b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa6cf968976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa6cf9f0610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa6cf9f0439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa6bcb84345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa6bcb8791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa6bcb87799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa6bcb62eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa6bfe3c6db start_thread at ??:0
@ 0x7fa6b8dd261f clone at ??:0
W20260620 23:08:17.324821 27265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:17.325129 27265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:17.325276 27265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:17.333175 27265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:17.333307 27265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:17.333465 27265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:08:17.343803 27265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:36499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=43467
--tserver_master_addrs=127.12.2.126:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:17.345078 27265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:17.346374 27265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:17.358771 27304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:17.358742 27305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:17.359061 27307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:17.359503 27265 server_base.cc:1034] running on GCE node
I20260620 23:08:17.360095 27265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:17.360949 27265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:17.362149 27265 hybrid_clock.cc:648] HybridClock initialized: now 1781996897362098 us; error 33 us; skew 500 ppm
I20260620 23:08:17.362416 27265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:17 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996897, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:17.365460 27265 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:08:17.366953 27265 webserver.cc:460] Webserver started at http://127.12.2.65:43467/ using document root <none> and password file <none>
I20260620 23:08:17.367664 27265 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:17.367790 27265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:17.373253 27265 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:08:17.376019 27314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:17.377341 27265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:08:17.377532 27265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "1b484a0c2cf44b0a8e3f52ef9291a968"
format_stamp: "Formatted at 2026-06-20 23:08:12 on dist-test-slave-7g8g"
server_key: "12af4a1da52b8fe7228c2eb5878a5b59"
server_key_iv: "e0fffb4b1adf40a818b433968afab846"
server_key_version: "encryptionkey@0"
I20260620 23:08:17.378518 27265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:17.392478 27265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:17.396260 27265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:17.396541 27265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:17.397497 27265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:17.398793 27265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:17.398891 27265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:17.399032 27265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:17.399117 27265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:17.430639 27265 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:36499
I20260620 23:08:17.430677 27426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:36499 every 8 connection(s)
I20260620 23:08:17.432204 27265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:08:17.433969 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 27265
I20260620 23:08:17.434554 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:38259
--local_ip_for_outbound_sockets=127.12.2.66
--tserver_master_addrs=127.12.2.126:36055
--webserver_port=33509
--webserver_interface=127.12.2.66
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:08:17.438131 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:17.436146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:48079)
0620 23:08:17.436967 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:17.437189 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":321,"threads_started":1}
W20260620 23:08:17.438854 27427 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:36055 (0 consecutive failures): Network error: Failed to ping master at 127.12.2.126:36055: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
I20260620 23:08:17.439750 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:17.439418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:52679)
0620 23:08:17.439574 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:17.439647 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 23:08:17.440629 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:17.440284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:55465)
0620 23:08:17.440431 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:17.440510 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 23:08:17.440863 27427 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:08:17.747983 27430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:17.748247 27430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:17.748369 27430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:17.755501 27430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:17.755604 27430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:17.755752 27430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:08:17.765221 27430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:38259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=33509
--tserver_master_addrs=127.12.2.126:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:17.766428 27430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:17.767726 27430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:17.779873 27435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:17.779939 27438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:17.779906 27436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:17.780782 27430 server_base.cc:1034] running on GCE node
I20260620 23:08:17.781313 27430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:17.782310 27430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:17.783527 27430 hybrid_clock.cc:648] HybridClock initialized: now 1781996897783456 us; error 36 us; skew 500 ppm
I20260620 23:08:17.783833 27430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:17 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996897, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:17.786998 27430 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:08:17.788326 27430 webserver.cc:460] Webserver started at http://127.12.2.66:33509/ using document root <none> and password file <none>
I20260620 23:08:17.788965 27430 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:17.789078 27430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:17.794203 27430 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:08:17.798607 27445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:17.799876 27430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:08:17.800021 27430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "83c3f45d312d41ee98c8c5bf4dbfb7ee"
format_stamp: "Formatted at 2026-06-20 23:08:12 on dist-test-slave-7g8g"
server_key: "2688aad9ef9ac6795b37a6f8937aad92"
server_key_iv: "3760f7740b9ee20dcd56a682e4247bef"
server_key_version: "encryptionkey@0"
I20260620 23:08:17.800971 27430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:17.820698 27430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:17.824254 27430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:17.824493 27430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:17.825358 27430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:17.826591 27430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:17.826671 27430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:17.826788 27430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:17.826853 27430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:17.855903 27430 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:38259
I20260620 23:08:17.855965 27557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:38259 every 8 connection(s)
I20260620 23:08:17.857324 27430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:08:17.861820 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:17.860093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:47637)
0620 23:08:17.860785 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:17.860983 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":254,"threads_started":1}
W20260620 23:08:17.862483 27558 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:36055 (0 consecutive failures): Network error: Failed to ping master at 127.12.2.126:36055: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
I20260620 23:08:17.863303 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:17.862912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:47627)
0620 23:08:17.863112 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:17.863191 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 23:08:17.864053 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:17.863769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:59541)
0620 23:08:17.863879 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:17.863952 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 23:08:17.864324 27558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:08:17.866652 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 27430
I20260620 23:08:17.867256 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:34451
--local_ip_for_outbound_sockets=127.12.2.67
--tserver_master_addrs=127.12.2.126:36055
--webserver_port=38433
--webserver_interface=127.12.2.67
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:18.178567 27561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:18.178846 27561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:18.179021 27561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:18.186287 27561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:18.186403 27561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:18.186556 27561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:08:18.196301 27561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:34451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=38433
--tserver_master_addrs=127.12.2.126:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:18.197603 27561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:18.198889 27561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:18.211529 27566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:18.211469 27569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:18.211447 27567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:18.212126 27561 server_base.cc:1034] running on GCE node
I20260620 23:08:18.212726 27561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:18.213609 27561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:18.214805 27561 hybrid_clock.cc:648] HybridClock initialized: now 1781996898214752 us; error 44 us; skew 500 ppm
I20260620 23:08:18.215163 27561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:18 dist-test-slave-7g8g krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996898, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:18.218189 27561 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:08:18.219609 27561 webserver.cc:460] Webserver started at http://127.12.2.67:38433/ using document root <none> and password file <none>
I20260620 23:08:18.220264 27561 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:18.220378 27561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:18.225760 27561 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 23:08:18.228340 27576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:18.229669 27561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:08:18.229840 27561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "6ee80df0b37c4d8aa6f4e70d9ee0943a"
format_stamp: "Formatted at 2026-06-20 23:08:13 on dist-test-slave-7g8g"
server_key: "b97be12e711a60e28eab4d6b6e6c3ee7"
server_key_iv: "4df8e949d9170130e0535e3bcde72cb5"
server_key_version: "encryptionkey@0"
I20260620 23:08:18.230868 27561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:18.247193 27561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:18.251101 27561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:18.251371 27561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:18.252353 27561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:18.253720 27561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:18.253824 27561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:18.253942 27561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:18.254020 27561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:18.285274 27561 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:34451
I20260620 23:08:18.285331 27688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:34451 every 8 connection(s)
I20260620 23:08:18.286656 27561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:08:18.287077 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 27561
I20260620 23:08:18.291500 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:18.289755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:42315)
0620 23:08:18.290420 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:18.290597 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":256,"threads_started":1}
W20260620 23:08:18.292114 27689 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:36055 (0 consecutive failures): Network error: Failed to ping master at 127.12.2.126:36055: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
I20260620 23:08:18.292865 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:18.292525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:55441)
0620 23:08:18.292686 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:18.292763 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:08:18.293635 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:18.293295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:43419)
0620 23:08:18.293439 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:18.293523 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 23:08:18.293874 27689 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:08:18.442181 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:18.441385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:60553)
0620 23:08:18.441833 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:18.441945 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":219,"threads_started":1}
I20260620 23:08:18.865485 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:18.864777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:41211)
0620 23:08:18.865187 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:18.865274 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":214,"threads_started":1}
I20260620 23:08:19.295354 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:19.294434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:47533)
0620 23:08:19.294966 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:19.295090 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":252,"threads_started":1}
I20260620 23:08:19.443686 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:19.443021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:39149)
0620 23:08:19.443440 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:19.443514 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":189,"threads_started":1}
I20260620 23:08:19.867012 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:19.866283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:55809)
0620 23:08:19.866743 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:19.866822 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":222,"threads_started":1}
I20260620 23:08:20.296990 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:20.296258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:52735)
0620 23:08:20.296751 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:20.296827 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":256,"threads_started":1}
I20260620 23:08:20.445199 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:20.444501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:40019)
0620 23:08:20.444932 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:20.445018 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260620 23:08:20.868695 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:20.867942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:50193)
0620 23:08:20.868427 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:20.868507 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":274,"threads_started":1}
I20260620 23:08:21.298552 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:21.297742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:43023)
0620 23:08:21.298250 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:21.298351 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
I20260620 23:08:21.446738 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:21.446055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:58953)
0620 23:08:21.446482 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:21.446561 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
I20260620 23:08:21.870276 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:21.869579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:56701)
0620 23:08:21.870005 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:21.870092 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":213,"threads_started":1}
I20260620 23:08:22.300155 27704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:22.299435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:44447)
0620 23:08:22.299864 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:22.299941 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":214,"threads_started":1}
I20260620 23:08:22.448449 27705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:22.447649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:38211)
0620 23:08:22.448180 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:22.448272 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":291,"threads_started":1}
I20260620 23:08:22.871882 27706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:22.871191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:39683)
0620 23:08:22.871639 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:22.871717 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260620 23:08:23.303328 27707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:23.302375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:49969)
0620 23:08:23.303062 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:23.303143 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":405,"threads_started":1}
I20260620 23:08:23.449954 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:23.449227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:55347)
0620 23:08:23.449684 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:23.449784 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":230,"threads_started":1}
I20260620 23:08:23.873601 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:23.872830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:37327)
0620 23:08:23.873306 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:23.873434 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":242,"threads_started":1}
I20260620 23:08:24.305156 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:24.304434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:54209)
0620 23:08:24.304893 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:24.304994 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":216,"threads_started":1}
I20260620 23:08:24.451584 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:24.450782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:44681)
0620 23:08:24.451288 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:24.451364 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":224,"threads_started":1}
I20260620 23:08:24.875281 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:24.874503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:34593)
0620 23:08:24.874983 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:24.875073 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":194,"threads_started":1}
I20260620 23:08:25.306790 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:25.305952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:36613)
0620 23:08:25.306463 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:25.306555 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":217,"threads_started":1}
I20260620 23:08:25.453054 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:25.452386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:34227)
0620 23:08:25.452798 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:25.452883 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":203,"threads_started":1}
I20260620 23:08:25.876895 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:25.876178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:57223)
0620 23:08:25.876637 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:25.876722 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
I20260620 23:08:26.308550 27716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:26.307752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:34539)
0620 23:08:26.308237 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:26.308337 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":256,"threads_started":1}
I20260620 23:08:26.454684 27717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:26.453950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:54031)
0620 23:08:26.454401 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:26.454480 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":217,"threads_started":1}
I20260620 23:08:26.878557 27718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:26.877735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:39117)
0620 23:08:26.878287 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:26.878374 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":276,"threads_started":1}
I20260620 23:08:27.310222 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:27.309487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:42761)
0620 23:08:27.309960 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:27.310048 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":232,"threads_started":1}
I20260620 23:08:27.456241 27720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:27.455525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:57423)
0620 23:08:27.455974 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:27.456058 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":227,"threads_started":1}
I20260620 23:08:27.880291 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:27.879542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:43575)
0620 23:08:27.880028 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:27.880123 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":247,"threads_started":1}
I20260620 23:08:28.311723 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:28.311085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:58803)
0620 23:08:28.311488 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:28.311567 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":214,"threads_started":1}
I20260620 23:08:28.457871 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:28.457143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:40567)
0620 23:08:28.457601 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:28.457692 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":216,"threads_started":1}
I20260620 23:08:28.882050 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:28.881226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:48221)
0620 23:08:28.881732 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:28.881837 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":237,"threads_started":1}
I20260620 23:08:29.313249 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:29.312578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:33995)
0620 23:08:29.312977 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:29.313076 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":199,"threads_started":1}
I20260620 23:08:29.459498 27726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:29.458771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:52825)
0620 23:08:29.459212 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:29.459294 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":241,"threads_started":1}
I20260620 23:08:29.883634 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:29.882964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:53943)
0620 23:08:29.883396 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:29.883476 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":196,"threads_started":1}
I20260620 23:08:30.314800 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:30.314144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:49107)
0620 23:08:30.314561 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:30.314644 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":211,"threads_started":1}
I20260620 23:08:30.461171 27729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:30.460376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:48399)
0620 23:08:30.460817 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:30.460971 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":206,"threads_started":1}
I20260620 23:08:30.885237 27730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:30.884443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:55809)
0620 23:08:30.884966 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:30.885056 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":240,"threads_started":1}
I20260620 23:08:31.316381 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:31.315654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:36891)
0620 23:08:31.316105 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:31.316196 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":206,"threads_started":1}
I20260620 23:08:31.462743 27732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:31.462047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:53141)
0620 23:08:31.462476 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:31.462558 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":223,"threads_started":1}
I20260620 23:08:31.886794 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:31.886123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:39163)
0620 23:08:31.886539 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:31.886618 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":211,"threads_started":1}
I20260620 23:08:32.317966 27734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:32.317241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:34511)
0620 23:08:32.317703 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:32.317787 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":242,"threads_started":1}
I20260620 23:08:32.464622 27735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:32.463878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.65:51371)
0620 23:08:32.464342 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:32.464439 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":232,"threads_started":1}
I20260620 23:08:32.888576 27736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:32.887861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.66:44815)
0620 23:08:32.888274 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:32.888380 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":211,"threads_started":1}
I20260620 23:08:33.291357 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 27265
I20260620 23:08:33.315321 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:33.292605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.12.2.65:36499)
0620 23:08:33.293175 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:33.293189 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:33.293229 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:08:33.294850 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:33.294860 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:33.295403 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:33.295784 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:33.297736 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:33.299189 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:33.300818 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:33.301960 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:33.302140 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:33.309219 (+ 7079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:33.309228 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:33.309245 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:33.309295 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:33.312216 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:33.312894 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:33.312900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:33.312907 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:33.313069 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:33.313439 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:33.313443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:33.313445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:33.314047 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:33.314299 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:33.314620 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:08:33.315030 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":273,"threads_started":1}
I20260620 23:08:33.316726 27422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:33.319937 27739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:33.319084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36055 (local address 127.12.2.67:59063)
0620 23:08:33.319582 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:33.319710 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":250,"threads_started":1}
I20260620 23:08:33.400134 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 27265
I20260620 23:08:33.414140 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 27430
I20260620 23:08:33.431121 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:33.415046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.12.2.66:38259)
0620 23:08:33.415509 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:33.415522 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:33.415559 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:08:33.416981 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:33.416989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:33.417429 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:33.417746 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:33.419157 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:33.420395 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:33.421600 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:33.422294 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:33.422471 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:33.425004 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:33.425012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:33.425028 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:33.425067 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:33.427864 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:33.428564 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:33.428571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:33.428577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:33.428742 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:33.429102 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:33.429105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:33.429107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:33.429639 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:33.429944 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:33.430368 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 23:08:33.430775 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":195,"threads_started":1}
I20260620 23:08:33.432523 27553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:33.513944 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 27430
I20260620 23:08:33.529956 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 27561
I20260620 23:08:33.547289 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:33.530893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.12.2.67:34451)
0620 23:08:33.531581 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:33.531594 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:33.531634 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:08:33.533179 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:33.533190 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:33.533634 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:33.533922 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:33.535410 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:33.536718 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:33.537995 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:33.538690 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:33.538826 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:33.541417 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:33.541429 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:33.541451 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:33.541498 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:33.544250 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:33.544857 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:33.544866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:33.544874 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:33.545031 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:33.545369 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:33.545375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:33.545377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:33.545863 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:33.546146 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:33.546582 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 23:08:33.547005 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":318,"threads_started":1}
I20260620 23:08:33.548673 27684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:33.629050 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 27561
2026-06-20T23:08:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23633 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:08:33 dist-test-slave-7g8g krb5kdc[27746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:08:33 dist-test-slave-7g8g krb5kdc[27746](info): set up 2 sockets
Jun 20 23:08:33 dist-test-slave-7g8g krb5kdc[27746](info): commencing operation
krb5kdc: starting...
W20260620 23:08:34.635288 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.007s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:08:34 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:08:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:08:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:08:34.855561 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43921
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:35.162400 27759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:35.162670 27759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:35.162783 27759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:35.169739 27759 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:08:35.169831 27759 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:08:35.169893 27759 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:08:35.169962 27759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:08:35.170025 27759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:35.170118 27759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:35.170172 27759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:08:35.170245 27759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:08:35.179920 27759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:43921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:35.181136 27759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:35.182420 27759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:35.193224 27764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:35.193307 27765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:35.193266 27767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:35.194355 27759 server_base.cc:1034] running on GCE node
I20260620 23:08:35.195135 27759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:35.196431 27759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:35.197631 27759 hybrid_clock.cc:648] HybridClock initialized: now 1781996915197572 us; error 32 us; skew 500 ppm
I20260620 23:08:35.197940 27759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:35 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:35.200821 27759 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:08:35.202184 27759 webserver.cc:460] Webserver started at http://127.12.2.126:34651/ using document root <none> and password file <none>
I20260620 23:08:35.202844 27759 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:35.202956 27759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:35.203260 27759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:35.206099 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "c174e249aa7a4dcb87c6e137fe9488b1"
format_stamp: "Formatted at 2026-06-20 23:08:35 on dist-test-slave-7g8g"
server_key: "f9278b3a1facb728c684f1cb05b4527c"
server_key_iv: "1ded91cf4e01ef1994974bf12f2edaa6"
server_key_version: "encryptionkey@0"
I20260620 23:08:35.206835 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "c174e249aa7a4dcb87c6e137fe9488b1"
format_stamp: "Formatted at 2026-06-20 23:08:35 on dist-test-slave-7g8g"
server_key: "f9278b3a1facb728c684f1cb05b4527c"
server_key_iv: "1ded91cf4e01ef1994974bf12f2edaa6"
server_key_version: "encryptionkey@0"
I20260620 23:08:35.211887 27759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 23:08:35.215260 27774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:35.216578 27759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:08:35.216737 27759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "c174e249aa7a4dcb87c6e137fe9488b1"
format_stamp: "Formatted at 2026-06-20 23:08:35 on dist-test-slave-7g8g"
server_key: "f9278b3a1facb728c684f1cb05b4527c"
server_key_iv: "1ded91cf4e01ef1994974bf12f2edaa6"
server_key_version: "encryptionkey@0"
I20260620 23:08:35.216938 27759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:35.237035 27759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:35.241030 27759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:35.241258 27759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:35.258383 27759 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:43921
I20260620 23:08:35.258390 27825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:43921 every 8 connection(s)
I20260620 23:08:35.259750 27759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:08:35.263358 27826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:35.265306 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 27759
I20260620 23:08:35.265487 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:08:35.266134 12297 external_mini_cluster.cc:1428] Setting key d30da11035869d02ecaedbe12f9e7856
I20260620 23:08:35.274442 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Bootstrap starting.
I20260620 23:08:35.277992 27826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:35.279255 27826 log.cc:824] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:08:35 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:35.281973 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: No bootstrap required, opened a new log
I20260620 23:08:35.288664 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:35.268019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.12.2.126:43921)
0620 23:08:35.268756 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:35.268768 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:35.268808 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:08:35.270402 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:35.270423 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:35.270805 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:35.271161 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:35.272879 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:35.274169 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:35.276042 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:35.276781 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:35.276930 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:35.281611 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:35.281619 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:35.281632 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:35.281671 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:35.284669 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:35.285365 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:35.285371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:35.285389 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:35.285551 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:35.285938 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:35.285941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:35.285943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:35.286491 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:35.286774 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:35.287264 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 23:08:35.287664 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":326,"threads_started":1}
I20260620 23:08:35.289539 27826 raft_consensus.cc:348] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:08:35.289882 27826 raft_consensus.cc:374] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:35.289994 27826 raft_consensus.cc:729] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c174e249aa7a4dcb87c6e137fe9488b1, State: Initialized, Role: FOLLOWER
I20260620 23:08:35.290553 27826 consensus_queue.cc:260] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:08:35.290786 27826 raft_consensus.cc:388] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:08:35.290908 27826 raft_consensus.cc:482] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:08:35.291129 27826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:35.293322 27826 raft_consensus.cc:504] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:08:35.293802 27826 leader_election.cc:302] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c174e249aa7a4dcb87c6e137fe9488b1; no voters:
I20260620 23:08:35.294471 27826 leader_election.cc:288] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:08:35.294595 27831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:08:35.295205 27831 raft_consensus.cc:686] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 1 LEADER]: Becoming Leader. State: Replica: c174e249aa7a4dcb87c6e137fe9488b1, State: Running, Role: LEADER
I20260620 23:08:35.295836 27831 consensus_queue.cc:237] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:08:35.297497 27826 sys_catalog.cc:564] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:08:35.301724 27832 sys_catalog.cc:455] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } } }
I20260620 23:08:35.302085 27832 sys_catalog.cc:458] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:35.303118 27833 sys_catalog.cc:455] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c174e249aa7a4dcb87c6e137fe9488b1. Latest consensus state: current_term: 1 leader_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } } }
I20260620 23:08:35.303424 27833 sys_catalog.cc:458] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:35.304065 27840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:08:35.309302 27840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:08:35.317782 27840 catalog_manager.cc:1347] Generated new cluster ID: 3d264e888cd643fc98af2e3d72741d3e
I20260620 23:08:35.317889 27840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:08:35.331319 27840 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:08:35.332679 27840 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:08:35.346191 27840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Generated new TSK 0
I20260620 23:08:35.346989 27840 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:08:35.415693 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43921
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:35.732851 27852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:35.733107 27852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:35.733196 27852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:35.740170 27852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:35.740270 27852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:35.740396 27852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:08:35.749893 27852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:35.751108 27852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:35.752314 27852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:35.764511 27858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:35.764492 27857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:35.764520 27860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:35.765337 27852 server_base.cc:1034] running on GCE node
I20260620 23:08:35.765924 27852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:35.766796 27852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:35.767998 27852 hybrid_clock.cc:648] HybridClock initialized: now 1781996915767955 us; error 47 us; skew 500 ppm
I20260620 23:08:35.768285 27852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:35 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:35.771314 27852 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:08:35.772648 27852 webserver.cc:460] Webserver started at http://127.12.2.65:43327/ using document root <none> and password file <none>
I20260620 23:08:35.773263 27852 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:35.773376 27852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:35.773653 27852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:35.777014 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "27f3b986af1940afad0f94e6b5b55756"
format_stamp: "Formatted at 2026-06-20 23:08:35 on dist-test-slave-7g8g"
server_key: "b10dbe8363f0ce1c43aababbc8d3ec8c"
server_key_iv: "f22f99f4d75f497488d38a10e7ad949b"
server_key_version: "encryptionkey@0"
I20260620 23:08:35.777973 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "27f3b986af1940afad0f94e6b5b55756"
format_stamp: "Formatted at 2026-06-20 23:08:35 on dist-test-slave-7g8g"
server_key: "b10dbe8363f0ce1c43aababbc8d3ec8c"
server_key_iv: "f22f99f4d75f497488d38a10e7ad949b"
server_key_version: "encryptionkey@0"
I20260620 23:08:35.782882 27852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:08:35.786369 27867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:35.787686 27852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:08:35.787832 27852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "27f3b986af1940afad0f94e6b5b55756"
format_stamp: "Formatted at 2026-06-20 23:08:35 on dist-test-slave-7g8g"
server_key: "b10dbe8363f0ce1c43aababbc8d3ec8c"
server_key_iv: "f22f99f4d75f497488d38a10e7ad949b"
server_key_version: "encryptionkey@0"
I20260620 23:08:35.788028 27852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:35.821529 27852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:35.825448 27852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:35.825696 27852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:35.826601 27852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:35.827906 27852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:35.827981 27852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:35.828101 27852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:35.828158 27852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:35.858508 27852 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:38291
I20260620 23:08:35.858603 27979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:38291 every 8 connection(s)
I20260620 23:08:35.859818 27852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:08:35.860781 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 27852
I20260620 23:08:35.861002 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:08:35.861799 12297 external_mini_cluster.cc:1428] Setting key 9b2794a949dae43669809091e2f9c6a6
Jun 20 23:08:35 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996915, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:35.880189 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:35.862588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:47677 (local address 127.12.2.126:43921)
0620 23:08:35.863076 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:35.863082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:35.864101 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 23:08:35.865894 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:35.865899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:35.865987 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:35.866089 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:35.867600 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:35.868617 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:35.870126 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:35.870688 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:35.870789 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:35.874913 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:35.874918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:35.874921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:35.874985 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:35.877305 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:35.877985 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:35.877990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:35.877992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:35.878106 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:35.878515 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:35.878518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:35.878520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:35.878774 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:35.878910 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:35.879604 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 23:08:35.879824 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":239,"threads_started":1}
I20260620 23:08:35.881186 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:35.863176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:47677)
0620 23:08:35.863938 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:35.863988 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:35.865638 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:35.866291 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:35.866299 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:35.866825 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:35.867427 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:35.867455 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:35.868767 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:35.868771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:35.869997 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:35.870010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:35.870993 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:35.871002 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:35.871312 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:35.872071 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:35.872114 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:35.874692 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:35.877445 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:35.877453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:35.877470 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:35.877865 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:35.878205 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:35.878210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:35.878214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:35.878406 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:35.878892 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:35.878900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:35.879239 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 23:08:35.879479 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 23:08:35.879999 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":323,"threads_started":1}
I20260620 23:08:35.882725 27980 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43921
I20260620 23:08:35.883044 27980 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
I20260620 23:08:35.883754 27980 heartbeater.cc:511] Master 127.12.2.126:43921 requested a full tablet report, sending...
I20260620 23:08:35.885926 27791 ts_manager.cc:194] Registered new tserver with Master: 27f3b986af1940afad0f94e6b5b55756 (127.12.2.65:38291)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
I20260620 23:08:35.887549 27791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:47677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:08:35.912201 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43921
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:36.220569 27986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:36.220834 27986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:36.220960 27986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:36.228068 27986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:36.228199 27986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:36.228359 27986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:08:36.238091 27986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:36.239387 27986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:36.240675 27986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:36.252912 27991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:36.252981 27992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:36.252936 27994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:36.253628 27986 server_base.cc:1034] running on GCE node
I20260620 23:08:36.254154 27986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:36.255066 27986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:36.256269 27986 hybrid_clock.cc:648] HybridClock initialized: now 1781996916256226 us; error 36 us; skew 500 ppm
I20260620 23:08:36.256569 27986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:36 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996916, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:36.259936 27986 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:08:36.264004 27986 webserver.cc:460] Webserver started at http://127.12.2.66:42581/ using document root <none> and password file <none>
I20260620 23:08:36.264632 27986 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:36.264734 27986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:36.265002 27986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:36.267709 27986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "010fc7adc8ee450197816f182dd0ac24"
format_stamp: "Formatted at 2026-06-20 23:08:36 on dist-test-slave-7g8g"
server_key: "b974840cd6f4c8aeb19f76e651cc7a05"
server_key_iv: "6c4701b5c4ebcc4d109e560f0bec9c62"
server_key_version: "encryptionkey@0"
I20260620 23:08:36.268421 27986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "010fc7adc8ee450197816f182dd0ac24"
format_stamp: "Formatted at 2026-06-20 23:08:36 on dist-test-slave-7g8g"
server_key: "b974840cd6f4c8aeb19f76e651cc7a05"
server_key_iv: "6c4701b5c4ebcc4d109e560f0bec9c62"
server_key_version: "encryptionkey@0"
I20260620 23:08:36.273599 27986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:08:36.276976 28001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:36.278223 27986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:08:36.278385 27986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "010fc7adc8ee450197816f182dd0ac24"
format_stamp: "Formatted at 2026-06-20 23:08:36 on dist-test-slave-7g8g"
server_key: "b974840cd6f4c8aeb19f76e651cc7a05"
server_key_iv: "6c4701b5c4ebcc4d109e560f0bec9c62"
server_key_version: "encryptionkey@0"
I20260620 23:08:36.278545 27986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:36.321684 27986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:36.325619 27986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:36.325886 27986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:36.326794 27986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:36.328050 27986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:36.328117 27986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:36.328220 27986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:36.328271 27986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:36.356488 27986 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:46477
I20260620 23:08:36.356549 28113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:46477 every 8 connection(s)
I20260620 23:08:36.357684 27986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:08:36.363561 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 27986
I20260620 23:08:36.363761 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:08:36.364323 12297 external_mini_cluster.cc:1428] Setting key 935eae26fcdee2849bb55ccc7be6502f
Jun 20 23:08:36 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996916, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:36.377382 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:36.360079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:56347 (local address 127.12.2.126:43921)
0620 23:08:36.360277 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:36.360283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:36.361345 (+ 1062us) server_negotiation.cc:407] Connection header received
0620 23:08:36.363167 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:36.363173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:36.363263 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:36.363346 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:36.365202 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.365970 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.367621 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.368231 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.368351 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:36.371720 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:36.371726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:36.371730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:36.371767 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:36.374464 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:36.375281 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:36.375284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:36.375287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:36.375423 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:36.375824 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:36.375827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:36.375829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:36.376070 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:36.376171 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:36.376957 (+ 786us) server_negotiation.cc:299] Negotiation successful
0620 23:08:36.377146 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:08:36.378683 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:36.360553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:56347)
0620 23:08:36.361197 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:36.361234 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:36.362868 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:36.363625 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:36.363638 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:36.364322 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:36.365052 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:36.365071 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.366114 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.366119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:36.367470 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:36.367482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.368415 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.368419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:36.368621 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:36.369205 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:36.369226 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:36.371500 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:36.374606 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:36.374616 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:36.374634 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:36.375179 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:36.375510 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:36.375515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:36.375519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:36.375735 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:36.376169 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:36.376177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:36.376609 (+ 432us) client_negotiation.cc:769] Sending connection context
0620 23:08:36.376876 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 23:08:36.377380 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":259,"threads_started":1}
I20260620 23:08:36.380162 28114 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43921
I20260620 23:08:36.380482 28114 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:36.381286 28114 heartbeater.cc:511] Master 127.12.2.126:43921 requested a full tablet report, sending...
I20260620 23:08:36.382824 27791 ts_manager.cc:194] Registered new tserver with Master: 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477)
I20260620 23:08:36.383683 27791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:56347
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:08:36.414137 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43921
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:36.716997 28119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:36.717284 28119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:36.717411 28119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:36.724495 28119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:36.724609 28119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:36.724761 28119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:08:36.734030 28119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:36.735267 28119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:36.736531 28119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:36.748536 28124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:36.748531 28125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:36.748659 28127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:36.749464 28119 server_base.cc:1034] running on GCE node
I20260620 23:08:36.749995 28119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:36.750878 28119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:36.752099 28119 hybrid_clock.cc:648] HybridClock initialized: now 1781996916752056 us; error 37 us; skew 500 ppm
I20260620 23:08:36.752375 28119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:36 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996916, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:36.755245 28119 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:08:36.756537 28119 webserver.cc:460] Webserver started at http://127.12.2.67:46151/ using document root <none> and password file <none>
I20260620 23:08:36.757162 28119 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:36.757272 28119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:36.757550 28119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:08:36.760274 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "131bc3e501ec4163b3918db0a43ddfb3"
format_stamp: "Formatted at 2026-06-20 23:08:36 on dist-test-slave-7g8g"
server_key: "58c3eaa3568c86f8ab9d53ef8a4b0815"
server_key_iv: "c26f0329923e14fdf7881fb729977af8"
server_key_version: "encryptionkey@0"
I20260620 23:08:36.760970 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "131bc3e501ec4163b3918db0a43ddfb3"
format_stamp: "Formatted at 2026-06-20 23:08:36 on dist-test-slave-7g8g"
server_key: "58c3eaa3568c86f8ab9d53ef8a4b0815"
server_key_iv: "c26f0329923e14fdf7881fb729977af8"
server_key_version: "encryptionkey@0"
I20260620 23:08:36.765733 28119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:08:36.768910 28134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:36.770118 28119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:08:36.770255 28119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "131bc3e501ec4163b3918db0a43ddfb3"
format_stamp: "Formatted at 2026-06-20 23:08:36 on dist-test-slave-7g8g"
server_key: "58c3eaa3568c86f8ab9d53ef8a4b0815"
server_key_iv: "c26f0329923e14fdf7881fb729977af8"
server_key_version: "encryptionkey@0"
I20260620 23:08:36.770443 28119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:36.792724 28119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:36.796460 28119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:36.796707 28119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:36.797650 28119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:36.798945 28119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:36.799029 28119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:36.799142 28119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:36.799192 28119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:36.827955 28119 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:40099
I20260620 23:08:36.828008 28246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:40099 every 8 connection(s)
I20260620 23:08:36.829149 28119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:08:36.834419 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 28119
I20260620 23:08:36.834642 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:08:36.835218 12297 external_mini_cluster.cc:1428] Setting key 72e9c0897ca6acd281b779c5a061223f
Jun 20 23:08:36 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996916, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:36.847604 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:36.831660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:37437 (local address 127.12.2.126:43921)
0620 23:08:36.831852 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:36.831858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:36.832986 (+ 1128us) server_negotiation.cc:407] Connection header received
0620 23:08:36.834985 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:36.834990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:36.835057 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:36.835140 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:36.836816 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.837691 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.839248 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.839638 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.839726 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:36.842611 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:36.842615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:36.842618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:36.842649 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:36.844999 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:36.845574 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:36.845577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:36.845580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:36.845695 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:36.846099 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:36.846103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:36.846105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:36.846381 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:36.846504 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:36.847199 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 23:08:36.847350 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:08:36.848644 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:36.832163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:37437)
0620 23:08:36.832823 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:36.832868 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:36.834670 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:36.835392 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:36.835403 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:36.835989 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:36.836659 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:36.836683 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.837827 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.837832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:36.839136 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:36.839148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.839789 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.839793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:36.839981 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:36.840515 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:36.840535 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:36.842447 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:36.845104 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:36.845110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:36.845121 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:36.845492 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:36.845767 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:36.845770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:36.845772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:36.845935 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:36.846487 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:36.846492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:36.846781 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 23:08:36.847112 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 23:08:36.847467 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":260,"threads_started":1}
I20260620 23:08:36.849910 28247 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43921
I20260620 23:08:36.850219 28247 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:36.850853 28247 heartbeater.cc:511] Master 127.12.2.126:43921 requested a full tablet report, sending...
I20260620 23:08:36.852255 27791 ts_manager.cc:194] Registered new tserver with Master: 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099)
I20260620 23:08:36.853104 27791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:37437
I20260620 23:08:36.862257 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:08:36.878381 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:36.866571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.12.2.126:43921)
0620 23:08:36.866750 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:36.866755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:36.867134 (+ 379us) server_negotiation.cc:407] Connection header received
0620 23:08:36.867284 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:36.867288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:36.867371 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:36.867447 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:36.868768 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.869477 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.870827 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.871269 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.871392 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:36.873714 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:36.873719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:36.873722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:36.873754 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:36.876236 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:36.876829 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:36.876833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:36.876835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:36.876954 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:36.877258 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:36.877260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:36.877262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:36.877572 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:36.877754 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:36.877977 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 23:08:36.878128 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:08:36.882009 27791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:08:36.888938 27791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:08:36.890504 27980 heartbeater.cc:503] Master 127.12.2.126:43921 was elected leader, sending a full tablet report...
I20260620 23:08:36.912192 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:36.900995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38291 (local address 127.0.0.1:56482)
0620 23:08:36.902693 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:36.902734 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:36.902906 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:36.903754 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:36.903762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:36.903805 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:36.904208 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:36.904219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.906618 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.906624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:36.909140 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:36.909157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.911169 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.911176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:36.911397 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:36.911455 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 23:08:36.911569 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:08:36.911760 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1481,"thread_start_us":1363,"threads_started":1}
I20260620 23:08:36.913612 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:36.901206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.12.2.65:38291)
0620 23:08:36.902189 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:36.902199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:36.903141 (+ 942us) server_negotiation.cc:407] Connection header received
0620 23:08:36.903249 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:36.903258 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:36.903534 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:36.904409 (+ 875us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:36.904687 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.906468 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.909369 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.910922 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.912725 (+ 1803us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:36.913004 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:36.913075 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 23:08:36.913254 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":1154,"threads_started":1}
I20260620 23:08:36.917903 27915 tablet_service.cc:1467] Processing CreateTablet for tablet e2d182d380e24e859b34b8a32526c776 (DEFAULT_TABLE table=test-table [id=72a4cd488c204d8ab3ad871a9b76c47c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:36.919480 27915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d182d380e24e859b34b8a32526c776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:36.921849 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:36.901621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:40099 (local address 127.0.0.1:35002)
0620 23:08:36.912455 (+ 10834us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:36.912472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:36.912559 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:36.915677 (+ 3118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:36.915681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:36.915703 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:36.915849 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:36.915858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.917979 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.917984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:36.919768 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:36.919780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.921077 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.921084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:36.921272 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:36.921304 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:08:36.921400 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:08:36.921528 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10739,"thread_start_us":13550,"threads_started":1}
I20260620 23:08:36.922528 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:36.902114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.12.2.67:40099)
0620 23:08:36.914448 (+ 12334us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:36.914459 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:36.914494 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:08:36.914596 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:36.914605 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:36.914872 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:36.916030 (+ 1158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:36.916303 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.917815 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.919937 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.920931 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.921053 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:36.921268 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:36.921446 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 23:08:36.921599 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12142,"thread_start_us":11968,"threads_started":1}
I20260620 23:08:36.924439 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:36.901713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:46477 (local address 127.0.0.1:35836)
0620 23:08:36.915426 (+ 13713us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:36.915443 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:36.915537 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:36.916891 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:36.916896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:36.916920 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:36.917319 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:36.917330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.919592 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.919597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:36.921965 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:36.921984 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.923708 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.923714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:36.923891 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:36.923924 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:08:36.924017 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:08:36.924138 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13591,"thread_start_us":15822,"threads_started":1}
I20260620 23:08:36.924521 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:36.902253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.12.2.66:46477)
0620 23:08:36.916116 (+ 13863us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:36.916124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:36.916151 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:08:36.916242 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:36.916249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:36.916501 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:36.916689 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:36.917950 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.919446 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.922148 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:36.923515 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:36.923642 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:36.923818 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:36.924068 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 23:08:36.924233 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13708,"thread_start_us":13617,"threads_started":1}
I20260620 23:08:36.925961 28182 tablet_service.cc:1467] Processing CreateTablet for tablet e2d182d380e24e859b34b8a32526c776 (DEFAULT_TABLE table=test-table [id=72a4cd488c204d8ab3ad871a9b76c47c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:36.927299 28182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d182d380e24e859b34b8a32526c776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:36.928076 28049 tablet_service.cc:1467] Processing CreateTablet for tablet e2d182d380e24e859b34b8a32526c776 (DEFAULT_TABLE table=test-table [id=72a4cd488c204d8ab3ad871a9b76c47c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:36.929397 28049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d182d380e24e859b34b8a32526c776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:36.934127 28265 tablet_bootstrap.cc:492] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: Bootstrap starting.
I20260620 23:08:36.937608 28265 tablet_bootstrap.cc:654] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:36.938803 28266 tablet_bootstrap.cc:492] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: Bootstrap starting.
I20260620 23:08:36.938848 28265 log.cc:824] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:36.942255 28266 tablet_bootstrap.cc:654] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:36.942425 28268 tablet_bootstrap.cc:492] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: Bootstrap starting.
I20260620 23:08:36.944787 28266 log.cc:824] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:36.945653 28268 tablet_bootstrap.cc:654] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:36.945744 28265 tablet_bootstrap.cc:492] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: No bootstrap required, opened a new log
I20260620 23:08:36.946067 28265 ts_tablet_manager.cc:1397] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260620 23:08:36.946803 28268 log.cc:824] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:36.947083 28266 tablet_bootstrap.cc:492] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: No bootstrap required, opened a new log
I20260620 23:08:36.947309 28266 ts_tablet_manager.cc:1397] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:08:36.949631 28268 tablet_bootstrap.cc:492] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: No bootstrap required, opened a new log
I20260620 23:08:36.949870 28268 ts_tablet_manager.cc:1397] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 23:08:36.954921 28266 raft_consensus.cc:348] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:36.955334 28266 raft_consensus.cc:374] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:36.955456 28266 raft_consensus.cc:729] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131bc3e501ec4163b3918db0a43ddfb3, State: Initialized, Role: FOLLOWER
I20260620 23:08:36.955991 28266 consensus_queue.cc:260] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:36.955859 28265 raft_consensus.cc:348] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:36.956248 28265 raft_consensus.cc:374] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:36.956341 28265 raft_consensus.cc:729] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f3b986af1940afad0f94e6b5b55756, State: Initialized, Role: FOLLOWER
I20260620 23:08:36.956941 28265 consensus_queue.cc:260] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:36.958693 28265 ts_tablet_manager.cc:1428] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 23:08:36.959661 28247 heartbeater.cc:503] Master 127.12.2.126:43921 was elected leader, sending a full tablet report...
I20260620 23:08:36.959694 28268 raft_consensus.cc:348] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:36.960095 28268 raft_consensus.cc:374] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:36.960188 28268 raft_consensus.cc:729] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010fc7adc8ee450197816f182dd0ac24, State: Initialized, Role: FOLLOWER
I20260620 23:08:36.960727 28268 consensus_queue.cc:260] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:36.961943 28266 ts_tablet_manager.cc:1428] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260620 23:08:36.962783 28114 heartbeater.cc:503] Master 127.12.2.126:43921 was elected leader, sending a full tablet report...
I20260620 23:08:36.963397 28268 ts_tablet_manager.cc:1428] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260620 23:08:37.071574 28272 raft_consensus.cc:482] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:08:37.071856 28272 raft_consensus.cc:504] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:37.073549 28272 leader_election.cc:288] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477), 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099)
I20260620 23:08:37.079905 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:37.074144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:50457 (local address 127.12.2.67:40099)
0620 23:08:37.074325 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:37.074331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:37.074955 (+ 624us) server_negotiation.cc:407] Connection header received
0620 23:08:37.075125 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:37.075128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:37.075186 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:37.075257 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:37.075862 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.076562 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.078258 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.079043 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.079150 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:37.079252 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:37.079533 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:08:37.079654 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:08:37.079955 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:37.073973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:58679 (local address 127.12.2.66:46477)
0620 23:08:37.074185 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:37.074190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:37.074858 (+ 668us) server_negotiation.cc:407] Connection header received
0620 23:08:37.075098 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:37.075102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:37.075165 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:37.075237 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:37.075815 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.076495 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.078216 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.079032 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.079156 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:37.079248 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:37.079569 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:08:37.079692 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:08:37.080065 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:37.074032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:40099 (local address 127.12.2.65:50457)
0620 23:08:37.074821 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:37.074839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:37.074990 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:37.075283 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:37.075287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:37.075326 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:37.075721 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:37.075730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.076696 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.076701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:37.078112 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:37.078123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.079253 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.079260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:37.079388 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:37.079410 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:08:37.079496 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:08:37.079619 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"spinlock_wait_cycles":1792,"thread_start_us":491,"threads_started":1}
I20260620 23:08:37.080103 28274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:37.073816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:46477 (local address 127.12.2.65:58679)
0620 23:08:37.074746 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:37.074764 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:37.074892 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:37.075267 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:37.075271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:37.075301 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:37.075683 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:37.075695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.076614 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.076618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:37.078072 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:37.078087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.079304 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.079309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:37.079432 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:37.079453 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:08:37.079532 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:08:37.079648 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":1,"thread_start_us":508,"threads_started":1}
W20260620 23:08:37.080637 28248 tablet.cc:2369] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:08:37.081295 28069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d182d380e24e859b34b8a32526c776" candidate_uuid: "27f3b986af1940afad0f94e6b5b55756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010fc7adc8ee450197816f182dd0ac24" is_pre_election: true
I20260620 23:08:37.081278 28202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d182d380e24e859b34b8a32526c776" candidate_uuid: "27f3b986af1940afad0f94e6b5b55756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "131bc3e501ec4163b3918db0a43ddfb3" is_pre_election: true
I20260620 23:08:37.081836 28069 raft_consensus.cc:2454] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f3b986af1940afad0f94e6b5b55756 in term 0.
I20260620 23:08:37.081836 28202 raft_consensus.cc:2454] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f3b986af1940afad0f94e6b5b55756 in term 0.
I20260620 23:08:37.082612 27869 leader_election.cc:302] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 131bc3e501ec4163b3918db0a43ddfb3, 27f3b986af1940afad0f94e6b5b55756; no voters:
I20260620 23:08:37.083063 28272 raft_consensus.cc:2781] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:08:37.083230 28272 raft_consensus.cc:482] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:08:37.083334 28272 raft_consensus.cc:3037] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:37.085536 28272 raft_consensus.cc:504] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:37.086156 28272 leader_election.cc:288] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 1 election: Requested vote from peers 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477), 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099)
I20260620 23:08:37.086796 28202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d182d380e24e859b34b8a32526c776" candidate_uuid: "27f3b986af1940afad0f94e6b5b55756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "131bc3e501ec4163b3918db0a43ddfb3"
I20260620 23:08:37.087037 28202 raft_consensus.cc:3037] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:37.087013 28069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d182d380e24e859b34b8a32526c776" candidate_uuid: "27f3b986af1940afad0f94e6b5b55756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010fc7adc8ee450197816f182dd0ac24"
I20260620 23:08:37.087212 28069 raft_consensus.cc:3037] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:37.089591 28069 raft_consensus.cc:2454] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3b986af1940afad0f94e6b5b55756 in term 1.
I20260620 23:08:37.090023 27868 leader_election.cc:302] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010fc7adc8ee450197816f182dd0ac24, 27f3b986af1940afad0f94e6b5b55756; no voters:
I20260620 23:08:37.090322 28272 raft_consensus.cc:2781] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:08:37.090785 28272 raft_consensus.cc:686] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 1 LEADER]: Becoming Leader. State: Replica: 27f3b986af1940afad0f94e6b5b55756, State: Running, Role: LEADER
I20260620 23:08:37.091591 28272 consensus_queue.cc:237] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:37.092233 28202 raft_consensus.cc:2454] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3b986af1940afad0f94e6b5b55756 in term 1.
I20260620 23:08:37.096392 27791 catalog_manager.cc:5495] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f3b986af1940afad0f94e6b5b55756 (127.12.2.65). New cstate: current_term: 1 leader_uuid: "27f3b986af1940afad0f94e6b5b55756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } health_report { overall_health: UNKNOWN } } }
W20260620 23:08:37.109606 28115 tablet.cc:2369] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:08:37.112213 27981 tablet.cc:2369] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:08:37.118783 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:37.111564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.12.2.65:38291)
0620 23:08:37.111756 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:37.111763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:37.111789 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:08:37.112064 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:37.112070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:37.112172 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:37.112329 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:08:37.112911 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.113960 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.115955 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.116561 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.116679 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:37.117240 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:08:37.117879 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:08:37.118023 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:37.118422 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 23:08:37.118513 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:08:37.127831 28202 raft_consensus.cc:1261] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3 [term 1 FOLLOWER]: Refusing update from remote peer 27f3b986af1940afad0f94e6b5b55756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:37.127929 28069 raft_consensus.cc:1261] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24 [term 1 FOLLOWER]: Refusing update from remote peer 27f3b986af1940afad0f94e6b5b55756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:37.128736 28276 consensus_queue.cc:1035] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:37.129097 28272 consensus_queue.cc:1035] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:37.157829 27791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:08:37.158116 27791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:08:37.161546 27791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Sending DeleteTablet for 3 replicas of tablet e2d182d380e24e859b34b8a32526c776
I20260620 23:08:37.162602 27915 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d182d380e24e859b34b8a32526c776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:37 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:56482
I20260620 23:08:37.162693 28049 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d182d380e24e859b34b8a32526c776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:37 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:35836
I20260620 23:08:37.163297 28182 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d182d380e24e859b34b8a32526c776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:37 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:35002
I20260620 23:08:37.163540 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 27852
I20260620 23:08:37.174356 28291 tablet_replica.cc:330] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: stopping tablet replica
I20260620 23:08:37.174912 28291 raft_consensus.cc:2229] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:08:37.175622 28291 raft_consensus.cc:2258] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:37.176518 28290 tablet_replica.cc:330] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: stopping tablet replica
I20260620 23:08:37.176900 28290 raft_consensus.cc:2229] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:08:37.177125 28289 tablet_replica.cc:330] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: stopping tablet replica
I20260620 23:08:37.177345 28290 raft_consensus.cc:2258] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:37.177604 28289 raft_consensus.cc:2229] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:08:37.178326 28289 raft_consensus.cc:2258] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:37.179462 28291 ts_tablet_manager.cc:1905] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:37.180164 28289 ts_tablet_manager.cc:1905] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:37.184262 28290 ts_tablet_manager.cc:1905] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:08:37 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:08:37.186203 28291 ts_tablet_manager.cc:1918] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:37.186358 28291 log.cc:1196] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/e2d182d380e24e859b34b8a32526c776
I20260620 23:08:37.186815 28291 ts_tablet_manager.cc:1939] T e2d182d380e24e859b34b8a32526c776 P 131bc3e501ec4163b3918db0a43ddfb3: Deleting consensus metadata
I20260620 23:08:37.188601 27776 catalog_manager.cc:4841] TS 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099): tablet e2d182d380e24e859b34b8a32526c776 (table test-table [id=72a4cd488c204d8ab3ad871a9b76c47c]) successfully deleted
I20260620 23:08:37.189236 28289 ts_tablet_manager.cc:1918] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:37.189419 28289 log.cc:1196] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/e2d182d380e24e859b34b8a32526c776
I20260620 23:08:37.190039 28289 ts_tablet_manager.cc:1939] T e2d182d380e24e859b34b8a32526c776 P 27f3b986af1940afad0f94e6b5b55756: Deleting consensus metadata
I20260620 23:08:37.192144 27777 catalog_manager.cc:4841] TS 27f3b986af1940afad0f94e6b5b55756 (127.12.2.65:38291): tablet e2d182d380e24e859b34b8a32526c776 (table test-table [id=72a4cd488c204d8ab3ad871a9b76c47c]) successfully deleted
I20260620 23:08:37.193279 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:37.164667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.12.2.65:38291)
0620 23:08:37.164873 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:37.164879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:37.173470 (+ 8591us) server_negotiation.cc:407] Connection header received
0620 23:08:37.173683 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:37.173689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:37.173800 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:37.173899 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:37.175991 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.177407 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.179705 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.180340 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.180463 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:37.185898 (+ 5435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:37.185912 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:37.185929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:37.185970 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:37.189789 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:37.190616 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:37.190625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:37.190629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:37.190809 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:37.191246 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:37.191251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:37.191253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:37.191689 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:37.192086 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:37.192695 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 23:08:37.192924 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:08:37.193982 27975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:37.195482 28290 ts_tablet_manager.cc:1918] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:37.195701 28290 log.cc:1196] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/e2d182d380e24e859b34b8a32526c776
I20260620 23:08:37.196514 28290 ts_tablet_manager.cc:1939] T e2d182d380e24e859b34b8a32526c776 P 010fc7adc8ee450197816f182dd0ac24: Deleting consensus metadata
I20260620 23:08:37.198415 27775 catalog_manager.cc:4841] TS 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477): tablet e2d182d380e24e859b34b8a32526c776 (table test-table [id=72a4cd488c204d8ab3ad871a9b76c47c]) successfully deleted
I20260620 23:08:37.288715 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 27852
I20260620 23:08:37.306200 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 27986
Jun 20 23:08:37 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:08:37.320529 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:37.306995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.12.2.66:46477)
0620 23:08:37.307185 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:37.307189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:37.307207 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:08:37.307351 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:37.307355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:37.307417 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:37.307500 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:37.308743 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.309524 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.310779 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.311264 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.311364 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:37.315157 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:37.315173 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:37.315188 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:37.315228 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:37.318110 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:37.318704 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:37.318711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:37.318714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:37.318851 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:37.319193 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:37.319196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:37.319198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:37.319592 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:37.319821 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:37.320085 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 23:08:37.320240 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:08:37.321161 28109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:37.407478 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 27986
I20260620 23:08:37.420563 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 28119
Jun 20 23:08:37 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:08:37.434544 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:37.421374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.12.2.67:40099)
0620 23:08:37.421610 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:37.421616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:37.421639 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:08:37.421806 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:37.421812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:37.421920 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:37.422028 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:37.423300 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.424033 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.425285 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:37.425768 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:37.425880 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:37.429850 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:37.429865 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:37.429894 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:37.429928 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:37.432386 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:37.432967 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:37.432974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:37.432976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:37.433112 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:37.433389 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:37.433393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:37.433395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:37.433719 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:37.433928 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:37.434137 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 23:08:37.434299 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:08:37.435133 28242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:37.521160 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 28119
I20260620 23:08:37.534001 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 27759
I20260620 23:08:37.535178 27821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:37.640899 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 27759
I20260620 23:08:57.658535 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43921
--webserver_interface=127.12.2.126
--webserver_port=34651
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:08:57.974542 28297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:57.974839 28297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:57.974989 28297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:57.982182 28297 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:08:57.982291 28297 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:08:57.982362 28297 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:08:57.982432 28297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:08:57.982506 28297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:57.982599 28297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:57.982659 28297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:08:57.982733 28297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:08:57.992622 28297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:43921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:57.993906 28297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:57.995227 28297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:58.016047 28303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:58.015985 28304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:58.016012 28306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:58.016575 28297 server_base.cc:1034] running on GCE node
I20260620 23:08:58.017556 28297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:58.019069 28297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:58.020284 28297 hybrid_clock.cc:648] HybridClock initialized: now 1781996938020205 us; error 47 us; skew 500 ppm
I20260620 23:08:58.020658 28297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:58 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:58.024026 28297 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:08:58.025472 28297 webserver.cc:460] Webserver started at http://127.12.2.126:34651/ using document root <none> and password file <none>
I20260620 23:08:58.026187 28297 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:58.026283 28297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:58.031911 28297 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:08:58.034624 28313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:58.035957 28297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:08:58.036113 28297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "c174e249aa7a4dcb87c6e137fe9488b1"
format_stamp: "Formatted at 2026-06-20 23:08:35 on dist-test-slave-7g8g"
server_key: "f9278b3a1facb728c684f1cb05b4527c"
server_key_iv: "1ded91cf4e01ef1994974bf12f2edaa6"
server_key_version: "encryptionkey@0"
I20260620 23:08:58.037091 28297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:58.067309 28297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:58.071372 28297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:58.071636 28297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:58.088570 28297 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:43921
I20260620 23:08:58.088600 28364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:43921 every 8 connection(s)
I20260620 23:08:58.090025 28297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:08:58.091010 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 28297
I20260620 23:08:58.091884 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:38291
--local_ip_for_outbound_sockets=127.12.2.65
--tserver_master_addrs=127.12.2.126:43921
--webserver_port=43327
--webserver_interface=127.12.2.65
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:08:58.097949 28365 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:08:58.101677 28365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Bootstrap starting.
I20260620 23:08:58.120728 28365 log.cc:824] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:58.132030 28365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:08:58.132545 28365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Bootstrap complete.
I20260620 23:08:58.141500 28365 raft_consensus.cc:348] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:08:58.142329 28365 raft_consensus.cc:729] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c174e249aa7a4dcb87c6e137fe9488b1, State: Initialized, Role: FOLLOWER
I20260620 23:08:58.142912 28365 consensus_queue.cc:260] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:08:58.143150 28365 raft_consensus.cc:388] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:08:58.143266 28365 raft_consensus.cc:482] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:08:58.143419 28365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:08:58.147164 28365 raft_consensus.cc:504] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:08:58.147655 28365 leader_election.cc:302] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c174e249aa7a4dcb87c6e137fe9488b1; no voters:
I20260620 23:08:58.148250 28365 leader_election.cc:288] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:08:58.148391 28369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:08:58.149437 28369 raft_consensus.cc:686] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 2 LEADER]: Becoming Leader. State: Replica: c174e249aa7a4dcb87c6e137fe9488b1, State: Running, Role: LEADER
I20260620 23:08:58.150027 28369 consensus_queue.cc:237] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:08:58.150404 28365 sys_catalog.cc:564] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:08:58.152580 28371 sys_catalog.cc:455] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c174e249aa7a4dcb87c6e137fe9488b1. Latest consensus state: current_term: 2 leader_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } } }
I20260620 23:08:58.152611 28370 sys_catalog.cc:455] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } } }
I20260620 23:08:58.152954 28371 sys_catalog.cc:458] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:58.152973 28370 sys_catalog.cc:458] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:08:58.164134 28376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:08:58.172216 28376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:08:58.173676 28376 catalog_manager.cc:1259] Loaded cluster ID: 3d264e888cd643fc98af2e3d72741d3e
I20260620 23:08:58.173830 28376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:08:58.177958 28376 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:08:58.179602 28376 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Loaded TSK: 0
I20260620 23:08:58.211833 28376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Generated new TSK 1
I20260620 23:08:58.212225 28376 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:08:58.427263 28367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:58.427520 28367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:58.427635 28367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:58.434767 28367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:58.434891 28367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:58.435071 28367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:08:58.444829 28367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:38291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=43327
--tserver_master_addrs=127.12.2.126:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:58.446089 28367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:58.447393 28367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:58.460244 28392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:58.460350 28395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:58.460435 28393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:58.461644 28367 server_base.cc:1034] running on GCE node
I20260620 23:08:58.462214 28367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:58.463227 28367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:58.464443 28367 hybrid_clock.cc:648] HybridClock initialized: now 1781996938464382 us; error 53 us; skew 500 ppm
I20260620 23:08:58.464747 28367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:58 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:58.472492 28367 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:08:58.473991 28367 webserver.cc:460] Webserver started at http://127.12.2.65:43327/ using document root <none> and password file <none>
I20260620 23:08:58.474676 28367 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:58.474799 28367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:58.480199 28367 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 23:08:58.482738 28402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:58.484052 28367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:08:58.484185 28367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "27f3b986af1940afad0f94e6b5b55756"
format_stamp: "Formatted at 2026-06-20 23:08:35 on dist-test-slave-7g8g"
server_key: "b10dbe8363f0ce1c43aababbc8d3ec8c"
server_key_iv: "f22f99f4d75f497488d38a10e7ad949b"
server_key_version: "encryptionkey@0"
I20260620 23:08:58.485106 28367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:58.505779 28367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:58.509841 28367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:58.510075 28367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:58.511085 28367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:58.512383 28367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:58.512465 28367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:58.512568 28367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:58.512637 28367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:58.542199 28367 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:38291
I20260620 23:08:58.542273 28514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:38291 every 8 connection(s)
I20260620 23:08:58.543594 28367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:08:58.543988 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 28367
I20260620 23:08:58.544669 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:46477
--local_ip_for_outbound_sockets=127.12.2.66
--tserver_master_addrs=127.12.2.126:43921
--webserver_port=42581
--webserver_interface=127.12.2.66
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:08:58 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:58.566466 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:58.546614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:50795)
0620 23:08:58.547455 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:58.547504 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:58.549158 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:58.550689 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:58.550697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:58.551306 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:58.551886 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:58.551904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:58.553515 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:58.553519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:58.554730 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:58.554742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:58.555766 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:58.555775 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:58.556007 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:58.556616 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:58.556642 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:58.558794 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:58.562238 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:58.562246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:58.562261 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:58.562692 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:58.563171 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:58.563176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:58.563180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:58.563359 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:58.564084 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:58.564091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:58.564398 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 23:08:58.564636 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 23:08:58.565173 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":351,"threads_started":1}
I20260620 23:08:58.566463 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:58.546429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:50795 (local address 127.12.2.126:43921)
0620 23:08:58.547017 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:58.547033 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:58.547631 (+ 598us) server_negotiation.cc:407] Connection header received
0620 23:08:58.549719 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:58.549737 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:58.550157 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:58.550546 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:58.552064 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:58.553361 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:58.554863 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:58.555539 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:58.555700 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:58.559057 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:58.559065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:58.559079 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:58.559120 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:58.562109 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:58.562824 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:58.562833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:58.562840 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:58.563066 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:58.563459 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:58.563464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:58.563468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:58.563921 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:58.564094 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:58.564851 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 23:08:58.565232 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":247,"threads_started":1}
I20260620 23:08:58.569304 28515 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43921
I20260620 23:08:58.569628 28515 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:58.570454 28515 heartbeater.cc:511] Master 127.12.2.126:43921 requested a full tablet report, sending...
I20260620 23:08:58.572777 28330 ts_manager.cc:194] Registered new tserver with Master: 27f3b986af1940afad0f94e6b5b55756 (127.12.2.65:38291)
I20260620 23:08:58.574283 28330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:50795
W20260620 23:08:58.857338 28519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:58.857589 28519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:58.857719 28519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:58.864810 28519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:58.864924 28519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:58.865077 28519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:08:58.874287 28519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:46477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=42581
--tserver_master_addrs=127.12.2.126:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:58.875496 28519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:58.876665 28519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:58.888396 28524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:58.888435 28525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:58.888468 28527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:58.889526 28519 server_base.cc:1034] running on GCE node
I20260620 23:08:58.890029 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:58.890825 28519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:58.892010 28519 hybrid_clock.cc:648] HybridClock initialized: now 1781996938891965 us; error 49 us; skew 500 ppm
I20260620 23:08:58.892279 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:58 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:58.895160 28519 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:08:58.896453 28519 webserver.cc:460] Webserver started at http://127.12.2.66:42581/ using document root <none> and password file <none>
I20260620 23:08:58.897059 28519 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:58.897166 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:58.902184 28519 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:08:58.904654 28534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:58.905874 28519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 23:08:58.905998 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "010fc7adc8ee450197816f182dd0ac24"
format_stamp: "Formatted at 2026-06-20 23:08:36 on dist-test-slave-7g8g"
server_key: "b974840cd6f4c8aeb19f76e651cc7a05"
server_key_iv: "6c4701b5c4ebcc4d109e560f0bec9c62"
server_key_version: "encryptionkey@0"
I20260620 23:08:58.906864 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:58.928155 28519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:58.932154 28519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:58.932368 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:58.933272 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:58.935153 28519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:58.935240 28519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:58.935351 28519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:58.935426 28519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:58.965000 28519 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:46477
I20260620 23:08:58.965071 28646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:46477 every 8 connection(s)
I20260620 23:08:58.966681 28519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:08:58.976259 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 28519
I20260620 23:08:58.976833 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:40099
--local_ip_for_outbound_sockets=127.12.2.67
--tserver_master_addrs=127.12.2.126:43921
--webserver_port=46151
--webserver_interface=127.12.2.67
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:08:58 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:58.985631 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:58.969086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:45403 (local address 127.12.2.126:43921)
0620 23:08:58.969265 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:58.969269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:58.970311 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 23:08:58.972143 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:58.972149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:58.972225 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:58.972313 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:58.973734 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:58.974425 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:58.975937 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:58.976340 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:58.976442 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:58.979902 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:58.979908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:58.979913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:58.979954 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:58.982726 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:58.983403 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:58.983407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:58.983410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:58.983545 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:58.983927 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:58.983931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:58.983934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:58.984269 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:58.984389 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:58.985195 (+ 806us) server_negotiation.cc:299] Negotiation successful
0620 23:08:58.985399 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:08:58.986881 28649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:58.969511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:45403)
0620 23:08:58.970158 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:58.970194 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:58.971873 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:58.972568 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:58.972576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:58.973045 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:58.973573 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:58.973589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:58.974568 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:58.974572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:58.975802 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:58.975814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:58.976668 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:58.976672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:58.976865 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:58.977503 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:58.977528 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:58.979669 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:58.982860 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:58.982866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:58.982878 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:58.983292 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:58.983644 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:58.983647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:58.983650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:58.983819 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:58.984387 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:58.984394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:58.984796 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 23:08:58.985103 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 23:08:58.985611 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":257,"threads_started":1}
I20260620 23:08:58.988432 28647 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43921
I20260620 23:08:58.988745 28647 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:58.989616 28647 heartbeater.cc:511] Master 127.12.2.126:43921 requested a full tablet report, sending...
I20260620 23:08:58.991140 28330 ts_manager.cc:194] Registered new tserver with Master: 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477)
I20260620 23:08:58.992044 28330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:45403
W20260620 23:08:59.288513 28650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:08:59.288793 28650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:08:59.288902 28650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:08:59.296406 28650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:08:59.296554 28650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:08:59.296801 28650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:08:59.308979 28650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:40099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=46151
--tserver_master_addrs=127.12.2.126:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:08:59.310264 28650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:08:59.311652 28650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:08:59.323642 28655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:59.323576 28656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:08:59.323619 28658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:08:59.324262 28650 server_base.cc:1034] running on GCE node
I20260620 23:08:59.324800 28650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:08:59.325646 28650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:08:59.326865 28650 hybrid_clock.cc:648] HybridClock initialized: now 1781996939326794 us; error 63 us; skew 500 ppm
I20260620 23:08:59.327195 28650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:08:59 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996939, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:08:59.330104 28650 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:08:59.331511 28650 webserver.cc:460] Webserver started at http://127.12.2.67:46151/ using document root <none> and password file <none>
I20260620 23:08:59.332150 28650 fs_manager.cc:359] Metadata directory not provided
I20260620 23:08:59.332273 28650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:08:59.337379 28650 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:08:59.339928 28665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:59.341228 28650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:08:59.341403 28650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "131bc3e501ec4163b3918db0a43ddfb3"
format_stamp: "Formatted at 2026-06-20 23:08:36 on dist-test-slave-7g8g"
server_key: "58c3eaa3568c86f8ab9d53ef8a4b0815"
server_key_iv: "c26f0329923e14fdf7881fb729977af8"
server_key_version: "encryptionkey@0"
I20260620 23:08:59.342341 28650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:08:59.359947 28650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:08:59.363843 28650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:08:59.364075 28650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:08:59.365047 28650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:08:59.366264 28650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:08:59.366361 28650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:59.366464 28650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:08:59.366536 28650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:08:59.396126 28650 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:40099
I20260620 23:08:59.396166 28777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:40099 every 8 connection(s)
I20260620 23:08:59.397491 28650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:08:59.400657 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 28650
Jun 20 23:08:59 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996939, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:08:59.416508 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.401540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38736 (local address 127.12.2.126:43921)
0620 23:08:59.401998 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.402004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.402308 (+ 304us) server_negotiation.cc:407] Connection header received
0620 23:08:59.402464 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.402469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.402548 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.402658 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:59.404353 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.405416 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.407336 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.407990 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.408118 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.411172 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:59.411177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:59.411182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:59.411212 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:59.413830 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:59.414482 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:59.414487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:59.414489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:59.414650 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:59.415121 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:59.415124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:59.415126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:59.415382 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:59.415583 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:59.415968 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 23:08:59.416142 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":230,"threads_started":1}
I20260620 23:08:59.419375 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.399873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:56319 (local address 127.12.2.126:43921)
0620 23:08:59.400051 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.400056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.401712 (+ 1656us) server_negotiation.cc:407] Connection header received
0620 23:08:59.403495 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.403501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.403575 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.403677 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:59.405470 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.406607 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.408679 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.409154 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.409276 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.413038 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:59.413043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:59.413046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:59.413092 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:59.416144 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:59.416989 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:59.416995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:59.416997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:59.417122 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:59.417643 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:59.417649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:59.417652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:59.417940 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:59.418065 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:59.418882 (+ 817us) server_negotiation.cc:299] Negotiation successful
0620 23:08:59.419123 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:08:59.420370 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.400337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:56319)
0620 23:08:59.401097 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:59.401146 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:59.403225 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:59.403854 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:59.403862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:59.404551 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:08:59.405276 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.405298 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.406791 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.406795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.408537 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.408549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.409407 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.409415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.409743 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.410360 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:08:59.410389 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:08:59.412811 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:08:59.416275 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:59.416285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:59.416302 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:59.416847 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:59.417283 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:08:59.417289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:08:59.417293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:08:59.417516 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:08:59.418061 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:08:59.418068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:08:59.418485 (+ 417us) client_negotiation.cc:769] Sending connection context
0620 23:08:59.418787 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 23:08:59.419312 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":318,"threads_started":1}
I20260620 23:08:59.421586 28778 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43921
I20260620 23:08:59.421875 28778 heartbeater.cc:464] Registering TS with master...
I20260620 23:08:59.422506 28778 heartbeater.cc:511] Master 127.12.2.126:43921 requested a full tablet report, sending...
I20260620 23:08:59.423739 28330 ts_manager.cc:194] Registered new tserver with Master: 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099)
I20260620 23:08:59.424475 28330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:56319
I20260620 23:08:59.431568 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:08:59.437952 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.432612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.12.2.126:43921)
0620 23:08:59.432785 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.432790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.433091 (+ 301us) server_negotiation.cc:407] Connection header received
0620 23:08:59.433284 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.433287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.433411 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.433529 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:08:59.433950 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.434662 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.436209 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.436629 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.436732 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.437195 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:08:59.437384 (+ 189us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 23:08:59.437708 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38742: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 23:08:59.438052 28517 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38742: token expired
W20260620 23:08:59.438637 28253 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.2.126:43921)
I20260620 23:08:59.450173 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.439060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.12.2.126:43921)
0620 23:08:59.439237 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.439241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.439257 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:08:59.439342 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.439345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.439397 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.439476 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:59.440704 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.441459 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.442665 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.443087 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.443173 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.445512 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:59.445516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:59.445520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:59.445571 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:59.447955 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:59.448603 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:59.448606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:59.448609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:59.448733 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:59.449030 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:59.449034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:59.449036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:59.449343 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:59.449515 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:59.449733 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:08:59.449878 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:08:59.453380 28330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:08:59.460768 28330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:08:59.483180 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.472603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:46477 (local address 127.0.0.1:52962)
0620 23:08:59.473555 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:59.473596 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:59.473765 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:59.476467 (+ 2702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:59.476474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:59.476501 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:59.476909 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.476939 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.478690 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.478696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.480594 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.480608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.482260 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.482266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.482457 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.482508 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:08:59.482640 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:08:59.482813 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":848,"threads_started":1}
I20260620 23:08:59.483311 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.473346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.12.2.66:46477)
0620 23:08:59.475814 (+ 2468us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.475823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.475858 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:08:59.475956 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.475963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.476221 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.476455 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:59.477092 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.478561 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.480748 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.482064 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.482183 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.482406 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:59.482801 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 23:08:59.482993 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2262,"thread_start_us":2163,"threads_started":1}
I20260620 23:08:59.483821 28791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.473267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.12.2.65:38291)
0620 23:08:59.476129 (+ 2862us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.476137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.476164 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:08:59.476260 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.476267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.476490 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.476685 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:59.477381 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.478810 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.481076 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.482406 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.482526 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.482701 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:59.483272 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 23:08:59.483480 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2701,"thread_start_us":2574,"threads_started":1}
I20260620 23:08:59.483842 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.472751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38291 (local address 127.0.0.1:36566)
0620 23:08:59.474898 (+ 2147us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:59.474916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:59.475088 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:59.476787 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:59.476792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:59.476814 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:59.477219 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.477232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.478950 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.478956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.480884 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.480898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.482609 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.482615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.482789 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.482822 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:08:59.482908 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:08:59.483047 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2009,"thread_start_us":1866,"threads_started":1}
I20260620 23:08:59.484567 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.473169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:40099 (local address 127.0.0.1:47572)
0620 23:08:59.476465 (+ 3296us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:59.476481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:59.476687 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:59.477412 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:59.477416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:59.477436 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:59.477754 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.477766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.479812 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.479817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.481820 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.481835 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.483857 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.483865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.484032 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.484062 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:08:59.484147 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:08:59.484274 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3168,"thread_start_us":3003,"threads_started":1}
I20260620 23:08:59.485119 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.473748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.12.2.67:40099)
0620 23:08:59.474423 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.474432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.476582 (+ 2150us) server_negotiation.cc:407] Connection header received
0620 23:08:59.476881 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.476889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.477119 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.477292 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:59.478113 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.479656 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.482003 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.483369 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.483484 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.483683 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:59.484674 (+ 991us) server_negotiation.cc:299] Negotiation successful
0620 23:08:59.484837 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":386,"threads_started":1}
I20260620 23:08:59.487232 28582 tablet_service.cc:1467] Processing CreateTablet for tablet b6ae3e7532994a7f88fcb300ab8cb8b9 (DEFAULT_TABLE table=test-table [id=e55dcf7035fe4d31bc730e5c439b0927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:59.487406 28450 tablet_service.cc:1467] Processing CreateTablet for tablet b6ae3e7532994a7f88fcb300ab8cb8b9 (DEFAULT_TABLE table=test-table [id=e55dcf7035fe4d31bc730e5c439b0927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:59.488478 28582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ae3e7532994a7f88fcb300ab8cb8b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:59.488720 28450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ae3e7532994a7f88fcb300ab8cb8b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:59.488615 28713 tablet_service.cc:1467] Processing CreateTablet for tablet b6ae3e7532994a7f88fcb300ab8cb8b9 (DEFAULT_TABLE table=test-table [id=e55dcf7035fe4d31bc730e5c439b0927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:08:59.489847 28713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ae3e7532994a7f88fcb300ab8cb8b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:08:59.499656 28794 tablet_bootstrap.cc:492] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: Bootstrap starting.
I20260620 23:08:59.502498 28794 tablet_bootstrap.cc:654] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:59.503671 28794 log.cc:824] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:59.507668 28795 tablet_bootstrap.cc:492] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Bootstrap starting.
I20260620 23:08:59.508684 28797 tablet_bootstrap.cc:492] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: Bootstrap starting.
I20260620 23:08:59.510098 28794 tablet_bootstrap.cc:492] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: No bootstrap required, opened a new log
I20260620 23:08:59.510340 28794 ts_tablet_manager.cc:1397] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 23:08:59.511919 28795 tablet_bootstrap.cc:654] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:59.511925 28797 tablet_bootstrap.cc:654] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: Neither blocks nor log segments found. Creating new log.
I20260620 23:08:59.513064 28797 log.cc:824] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:59.513064 28795 log.cc:824] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Log is configured to *not* fsync() on all Append() calls
I20260620 23:08:59.515517 28797 tablet_bootstrap.cc:492] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: No bootstrap required, opened a new log
I20260620 23:08:59.515731 28797 ts_tablet_manager.cc:1397] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 23:08:59.515913 28795 tablet_bootstrap.cc:492] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: No bootstrap required, opened a new log
I20260620 23:08:59.516158 28795 ts_tablet_manager.cc:1397] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 23:08:59.519686 28794 raft_consensus.cc:348] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:59.520090 28794 raft_consensus.cc:374] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:59.520183 28794 raft_consensus.cc:729] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010fc7adc8ee450197816f182dd0ac24, State: Initialized, Role: FOLLOWER
I20260620 23:08:59.520774 28794 consensus_queue.cc:260] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:59.521811 28647 heartbeater.cc:503] Master 127.12.2.126:43921 was elected leader, sending a full tablet report...
I20260620 23:08:59.523089 28794 ts_tablet_manager.cc:1428] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260620 23:08:59.523520 28797 raft_consensus.cc:348] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:59.523909 28797 raft_consensus.cc:374] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:59.524029 28797 raft_consensus.cc:729] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131bc3e501ec4163b3918db0a43ddfb3, State: Initialized, Role: FOLLOWER
I20260620 23:08:59.524590 28797 consensus_queue.cc:260] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:59.525593 28778 heartbeater.cc:503] Master 127.12.2.126:43921 was elected leader, sending a full tablet report...
I20260620 23:08:59.525935 28795 raft_consensus.cc:348] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:59.526202 28797 ts_tablet_manager.cc:1428] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 23:08:59.526319 28795 raft_consensus.cc:374] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:08:59.526412 28795 raft_consensus.cc:729] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f3b986af1940afad0f94e6b5b55756, State: Initialized, Role: FOLLOWER
I20260620 23:08:59.526971 28795 consensus_queue.cc:260] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:59.527937 28515 heartbeater.cc:503] Master 127.12.2.126:43921 was elected leader, sending a full tablet report...
I20260620 23:08:59.528519 28795 ts_tablet_manager.cc:1428] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
W20260620 23:08:59.545676 28516 tablet.cc:2369] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:08:59.648910 28779 tablet.cc:2369] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:08:59.694576 28800 raft_consensus.cc:482] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:08:59.694844 28800 raft_consensus.cc:504] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:59.696435 28800 leader_election.cc:288] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f3b986af1940afad0f94e6b5b55756 (127.12.2.65:38291), 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099)
I20260620 23:08:59.705334 28791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.696929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:47029 (local address 127.12.2.65:38291)
0620 23:08:59.697122 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.697128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.697812 (+ 684us) server_negotiation.cc:407] Connection header received
0620 23:08:59.697998 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.698004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.698080 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.698169 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:59.698968 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.700025 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.702887 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.704193 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.704353 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.704460 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:59.704908 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 23:08:59.705044 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:08:59.705344 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.696934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:45067 (local address 127.12.2.67:40099)
0620 23:08:59.697121 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.697127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.697813 (+ 686us) server_negotiation.cc:407] Connection header received
0620 23:08:59.698004 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.698008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.698080 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.698169 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:08:59.698892 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.700025 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.702882 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.704227 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.704352 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.704466 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:59.704908 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 23:08:59.705044 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:08:59.705492 28803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.696858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38291 (local address 127.12.2.66:47029)
0620 23:08:59.697678 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:59.697697 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:59.697863 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:59.698200 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:59.698205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:59.698232 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:59.698822 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.698833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.700175 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.700179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.702708 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.702724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.704462 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.704467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.704685 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.704783 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 23:08:59.704874 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 23:08:59.705050 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":245,"thread_start_us":466,"threads_started":1}
I20260620 23:08:59.705749 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.696858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:40099 (local address 127.12.2.66:45067)
0620 23:08:59.697728 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:59.697742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:08:59.697901 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:08:59.698200 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:08:59.698205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:08:59.698244 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:08:59.698722 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.698735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.700173 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.700179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.702708 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:08:59.702724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.704460 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.704467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:08:59.704689 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.704782 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 23:08:59.704874 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:08:59.705050 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":284,"spinlock_wait_cycles":52224,"thread_start_us":428,"threads_started":1}
I20260620 23:08:59.706599 28470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae3e7532994a7f88fcb300ab8cb8b9" candidate_uuid: "010fc7adc8ee450197816f182dd0ac24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f3b986af1940afad0f94e6b5b55756" is_pre_election: true
I20260620 23:08:59.706691 28733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae3e7532994a7f88fcb300ab8cb8b9" candidate_uuid: "010fc7adc8ee450197816f182dd0ac24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "131bc3e501ec4163b3918db0a43ddfb3" is_pre_election: true
I20260620 23:08:59.707173 28470 raft_consensus.cc:2454] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 010fc7adc8ee450197816f182dd0ac24 in term 0.
I20260620 23:08:59.707238 28733 raft_consensus.cc:2454] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 010fc7adc8ee450197816f182dd0ac24 in term 0.
I20260620 23:08:59.707959 28536 leader_election.cc:302] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 010fc7adc8ee450197816f182dd0ac24, 131bc3e501ec4163b3918db0a43ddfb3, 27f3b986af1940afad0f94e6b5b55756; no voters:
I20260620 23:08:59.708402 28800 raft_consensus.cc:2781] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:08:59.708530 28800 raft_consensus.cc:482] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:08:59.708626 28800 raft_consensus.cc:3037] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:59.710762 28800 raft_consensus.cc:504] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:08:59.711365 28800 leader_election.cc:288] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [CANDIDATE]: Term 1 election: Requested vote from peers 27f3b986af1940afad0f94e6b5b55756 (127.12.2.65:38291), 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099)
I20260620 23:08:59.711786 28470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae3e7532994a7f88fcb300ab8cb8b9" candidate_uuid: "010fc7adc8ee450197816f182dd0ac24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f3b986af1940afad0f94e6b5b55756"
I20260620 23:08:59.711819 28733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae3e7532994a7f88fcb300ab8cb8b9" candidate_uuid: "010fc7adc8ee450197816f182dd0ac24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "131bc3e501ec4163b3918db0a43ddfb3"
I20260620 23:08:59.711956 28470 raft_consensus.cc:3037] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:59.711995 28733 raft_consensus.cc:3037] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:08:59.714030 28470 raft_consensus.cc:2454] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 010fc7adc8ee450197816f182dd0ac24 in term 1.
I20260620 23:08:59.714069 28733 raft_consensus.cc:2454] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 010fc7adc8ee450197816f182dd0ac24 in term 1.
I20260620 23:08:59.714425 28537 leader_election.cc:302] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010fc7adc8ee450197816f182dd0ac24, 27f3b986af1940afad0f94e6b5b55756; no voters:
I20260620 23:08:59.714725 28800 raft_consensus.cc:2781] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:08:59.715152 28800 raft_consensus.cc:686] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 1 LEADER]: Becoming Leader. State: Replica: 010fc7adc8ee450197816f182dd0ac24, State: Running, Role: LEADER
I20260620 23:08:59.715595 28800 consensus_queue.cc:237] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
W20260620 23:08:59.718410 28648 tablet.cc:2369] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:08:59.720172 28330 catalog_manager.cc:5495] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66). New cstate: current_term: 1 leader_uuid: "010fc7adc8ee450197816f182dd0ac24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } health_report { overall_health: UNKNOWN } } }
I20260620 23:08:59.778450 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.771672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.12.2.66:46477)
0620 23:08:59.771923 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.771929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.772010 (+ 81us) server_negotiation.cc:407] Connection header received
0620 23:08:59.772290 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.772296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.772406 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.772550 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:08:59.772991 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.774033 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.776010 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.776523 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.776640 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.777326 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:08:59.777929 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:08:59.778032 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:59.778154 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 23:08:59.778219 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:08:59.786382 28470 raft_consensus.cc:1261] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Refusing update from remote peer 010fc7adc8ee450197816f182dd0ac24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:59.786391 28733 raft_consensus.cc:1261] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3 [term 1 FOLLOWER]: Refusing update from remote peer 010fc7adc8ee450197816f182dd0ac24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:08:59.787271 28805 consensus_queue.cc:1035] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:59.787571 28800 consensus_queue.cc:1035] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:08:59.797537 28808 mvcc.cc:205] Tried to move back new op lower bound from 7299059465355460608 to 7299059465078726656. Current Snapshot: MvccSnapshot[applied={T|T < 7299059465355460608}]
I20260620 23:08:59.798187 28809 mvcc.cc:205] Tried to move back new op lower bound from 7299059465355460608 to 7299059465078726656. Current Snapshot: MvccSnapshot[applied={T|T < 7299059465355460608}]
I20260620 23:08:59.801152 28810 mvcc.cc:205] Tried to move back new op lower bound from 7299059465355460608 to 7299059465078726656. Current Snapshot: MvccSnapshot[applied={T|T < 7299059465355460608}]
I20260620 23:08:59.811462 28330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:08:59.811731 28330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:08:59.815232 28330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Sending DeleteTablet for 3 replicas of tablet b6ae3e7532994a7f88fcb300ab8cb8b9
I20260620 23:08:59.816275 28582 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ae3e7532994a7f88fcb300ab8cb8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:59 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:52962
I20260620 23:08:59.816548 28713 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ae3e7532994a7f88fcb300ab8cb8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:59 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:47572
I20260620 23:08:59.816552 28450 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ae3e7532994a7f88fcb300ab8cb8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:08:59 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:36566
I20260620 23:08:59.817800 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 28367
I20260620 23:08:59.828573 28817 tablet_replica.cc:330] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: stopping tablet replica
I20260620 23:08:59.829092 28817 raft_consensus.cc:2229] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:08:59.830010 28817 raft_consensus.cc:2258] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:59.832247 28817 ts_tablet_manager.cc:1905] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:59.837374 28791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.818636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.12.2.65:38291)
0620 23:08:59.818812 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.818830 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.818903 (+ 73us) server_negotiation.cc:407] Connection header received
0620 23:08:59.819125 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.819131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.819216 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.819324 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:59.821075 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.822383 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.824366 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.825078 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.825205 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.830094 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:59.830115 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:59.830135 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:59.830179 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:59.834155 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:59.834985 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:59.834995 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:59.834999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:59.835166 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:59.835566 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:59.835571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:59.835575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:59.836107 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:59.836406 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:59.836695 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 23:08:59.836977 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:08:59.838188 28510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:08:59.839637 28817 ts_tablet_manager.cc:1918] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:59.839855 28817 log.cc:1196] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/b6ae3e7532994a7f88fcb300ab8cb8b9
I20260620 23:08:59.840519 28817 ts_tablet_manager.cc:1939] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 010fc7adc8ee450197816f182dd0ac24: Deleting consensus metadata
I20260620 23:08:59.842530 28314 catalog_manager.cc:4841] TS 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477): tablet b6ae3e7532994a7f88fcb300ab8cb8b9 (table test-table [id=e55dcf7035fe4d31bc730e5c439b0927]) successfully deleted
I20260620 23:08:59.844650 28818 tablet_replica.cc:330] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: stopping tablet replica
I20260620 23:08:59.845176 28818 raft_consensus.cc:2229] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:08:59.845816 28818 raft_consensus.cc:2258] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:08:59.847584 28818 ts_tablet_manager.cc:1905] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:08:59.852365 28818 ts_tablet_manager.cc:1918] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:08:59.852504 28818 log.cc:1196] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/b6ae3e7532994a7f88fcb300ab8cb8b9
I20260620 23:08:59.852913 28818 ts_tablet_manager.cc:1939] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 131bc3e501ec4163b3918db0a43ddfb3: Deleting consensus metadata
I20260620 23:08:59.854583 28315 catalog_manager.cc:4841] TS 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099): tablet b6ae3e7532994a7f88fcb300ab8cb8b9 (table test-table [id=e55dcf7035fe4d31bc730e5c439b0927]) successfully deleted
I20260620 23:08:59.952868 28819 tablet_replica.cc:330] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: stopping tablet replica
I20260620 23:08:59.953388 28819 raft_consensus.cc:2229] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:08:59.953450 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 28367
W20260620 23:08:59.968932 28316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.2.65:38291 (error 108)
I20260620 23:08:59.970690 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.970156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38291 (local address 127.0.0.1:36582)
0620 23:08:59.970421 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 23:08:59.970527 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:38291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260620 23:08:59.970992 28316 catalog_manager.cc:4584] TS 27f3b986af1940afad0f94e6b5b55756 (127.12.2.65:38291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6ae3e7532994a7f88fcb300ab8cb8b9: Network error: Client connection negotiation failed: client connection to 127.12.2.65:38291: connect: Connection refused (error 111)
I20260620 23:08:59.971259 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 28519
I20260620 23:08:59.988679 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:08:59.972023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.12.2.66:46477)
0620 23:08:59.972191 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:08:59.972197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:08:59.972322 (+ 125us) server_negotiation.cc:407] Connection header received
0620 23:08:59.972480 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:08:59.972486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:08:59.972575 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:08:59.972681 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:08:59.974418 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.975708 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.977572 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:08:59.978272 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:08:59.978402 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:08:59.981506 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:08:59.981519 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:08:59.981539 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:08:59.981601 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:08:59.985648 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:59.986466 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:59.986479 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:59.986482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:59.986658 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:08:59.987057 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:08:59.987062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:08:59.987066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:08:59.987524 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:08:59.987850 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 23:08:59.988089 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 23:08:59.988302 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:08:59.989355 28642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:00.024523 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:00.024055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38291 (local address 127.0.0.1:36596)
0620 23:09:00.024263 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:00.024382 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:38291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:09:00.025481 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:00.025160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38291 (local address 127.0.0.1:36602)
0620 23:09:00.025284 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:00.025367 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:38291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 23:09:00.086053 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:00.085583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38291 (local address 127.0.0.1:36616)
0620 23:09:00.085793 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:00.085888 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:38291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:09:00.087069 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:00.086717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38291 (local address 127.0.0.1:36622)
0620 23:09:00.086846 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:00.086915 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:38291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:09:00.101132 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 28519
I20260620 23:09:00.116377 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 28650
I20260620 23:09:00.128575 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:00.117063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.12.2.67:40099)
0620 23:09:00.117278 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:00.117283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:00.117302 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:09:00.117421 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:00.117425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:00.117487 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:00.117566 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:00.118802 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:00.119560 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:00.120838 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:00.121188 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:00.121285 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:00.123639 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:00.123652 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:00.123669 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:00.123714 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:00.126318 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:00.126879 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:00.126885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:00.126887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:00.127046 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:00.127335 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:00.127338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:00.127340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:00.127668 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:00.127869 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:00.128095 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 23:09:00.128254 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:09:00.129185 28773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:00.167717 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:00.167235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38291 (local address 127.0.0.1:36630)
0620 23:09:00.167437 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:00.167551 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:38291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 23:09:00.168712 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:00.168337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:38291 (local address 127.0.0.1:36644)
0620 23:09:00.168522 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:00.168594 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:38291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 23:09:00.215323 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 28650
I20260620 23:09:00.228005 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 28297
I20260620 23:09:00.229107 28360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:00.341771 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 28297
I20260620 23:09:20.359442 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43921
--webserver_interface=127.12.2.126
--webserver_port=34651
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:20.681176 28825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:20.681460 28825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:20.681578 28825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:20.688676 28825 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:09:20.688766 28825 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:09:20.688807 28825 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:09:20.688874 28825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:09:20.688941 28825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:20.689034 28825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:20.689088 28825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:09:20.689162 28825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:09:20.698860 28825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:43921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:20.700135 28825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:20.701443 28825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:20.712841 28830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:20.712893 28833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:20.712924 28831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:20.713491 28825 server_base.cc:1034] running on GCE node
I20260620 23:09:20.714174 28825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:20.715538 28825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:20.716749 28825 hybrid_clock.cc:648] HybridClock initialized: now 1781996960716692 us; error 25 us; skew 500 ppm
I20260620 23:09:20.717078 28825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:20 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996960, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:20.720079 28825 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:09:20.721490 28825 webserver.cc:460] Webserver started at http://127.12.2.126:34651/ using document root <none> and password file <none>
I20260620 23:09:20.722191 28825 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:20.722286 28825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:20.727630 28825 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:09:20.730372 28840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:20.731737 28825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:09:20.731894 28825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "c174e249aa7a4dcb87c6e137fe9488b1"
format_stamp: "Formatted at 2026-06-20 23:08:35 on dist-test-slave-7g8g"
server_key: "f9278b3a1facb728c684f1cb05b4527c"
server_key_iv: "1ded91cf4e01ef1994974bf12f2edaa6"
server_key_version: "encryptionkey@0"
I20260620 23:09:20.732815 28825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:20.758003 28825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:20.762068 28825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:20.762336 28825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:20.779615 28825 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:43921
I20260620 23:09:20.779647 28891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:43921 every 8 connection(s)
I20260620 23:09:20.781023 28825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:09:20.782692 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 28825
I20260620 23:09:20.783761 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:38291
--local_ip_for_outbound_sockets=127.12.2.65
--tserver_master_addrs=127.12.2.126:43921
--webserver_port=43327
--webserver_interface=127.12.2.65
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:09:20.789979 28892 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:09:20.794001 28892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Bootstrap starting.
I20260620 23:09:20.813885 28892 log.cc:824] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:20.829201 28892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:09:20.829702 28892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Bootstrap complete.
I20260620 23:09:20.838590 28892 raft_consensus.cc:348] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:09:20.839484 28892 raft_consensus.cc:729] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c174e249aa7a4dcb87c6e137fe9488b1, State: Initialized, Role: FOLLOWER
I20260620 23:09:20.840206 28892 consensus_queue.cc:260] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:09:20.840415 28892 raft_consensus.cc:388] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:09:20.840528 28892 raft_consensus.cc:482] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:09:20.840689 28892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:09:20.844441 28892 raft_consensus.cc:504] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:09:20.844892 28892 leader_election.cc:302] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c174e249aa7a4dcb87c6e137fe9488b1; no voters:
I20260620 23:09:20.845494 28892 leader_election.cc:288] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:09:20.845638 28896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:09:20.846594 28896 raft_consensus.cc:686] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [term 3 LEADER]: Becoming Leader. State: Replica: c174e249aa7a4dcb87c6e137fe9488b1, State: Running, Role: LEADER
I20260620 23:09:20.847191 28896 consensus_queue.cc:237] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } }
I20260620 23:09:20.847571 28892 sys_catalog.cc:564] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:09:20.849560 28898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c174e249aa7a4dcb87c6e137fe9488b1. Latest consensus state: current_term: 3 leader_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } } }
I20260620 23:09:20.849900 28898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:20.850415 28897 sys_catalog.cc:455] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c174e249aa7a4dcb87c6e137fe9488b1" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43921 } } }
I20260620 23:09:20.850715 28897 sys_catalog.cc:458] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:20.859193 28901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:09:20.868254 28901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:09:20.870126 28901 catalog_manager.cc:1259] Loaded cluster ID: 3d264e888cd643fc98af2e3d72741d3e
I20260620 23:09:20.870231 28901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:09:20.874236 28901 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:09:20.875844 28901 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Loaded TSK: 0
I20260620 23:09:20.876077 28901 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c174e249aa7a4dcb87c6e137fe9488b1: Loaded TSK: 1
F20260620 23:09:20.877305 28901 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe26cebb9ad google::LogMessage::Fail() at ??:0
@ 0x7fe26cebe88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe26cebb4de google::LogMessage::Flush() at ??:0
@ 0x7fe26cebca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe270c1f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe270c0e73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe2828ea1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe2828e9783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe2829752a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe2829750c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe2829bf5e9 std::function<>::operator()() at ??:0
@ 0x7fe2828edcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe2828ebc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe282973610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe282973439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe26fb07345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe26fb0a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe26fb0a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe26fae5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe272dbf6db start_thread at ??:0
@ 0x7fe26bd5561f clone at ??:0
W20260620 23:09:21.205067 28894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:21.205330 28894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:21.205422 28894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:21.212512 28894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:21.212625 28894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:21.212744 28894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:09:21.222203 28894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:38291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=43327
--tserver_master_addrs=127.12.2.126:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:21.223495 28894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:21.224762 28894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:21.237063 28939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:21.236980 28940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:21.237074 28942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:21.238003 28894 server_base.cc:1034] running on GCE node
I20260620 23:09:21.238582 28894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:21.239486 28894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:21.240686 28894 hybrid_clock.cc:648] HybridClock initialized: now 1781996961240642 us; error 28 us; skew 500 ppm
I20260620 23:09:21.240990 28894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:21 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996961, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:21.243865 28894 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:09:21.245219 28894 webserver.cc:460] Webserver started at http://127.12.2.65:43327/ using document root <none> and password file <none>
I20260620 23:09:21.245857 28894 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:21.245961 28894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:21.251052 28894 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:09:21.253724 28949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:21.254990 28894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:09:21.255125 28894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "27f3b986af1940afad0f94e6b5b55756"
format_stamp: "Formatted at 2026-06-20 23:08:35 on dist-test-slave-7g8g"
server_key: "b10dbe8363f0ce1c43aababbc8d3ec8c"
server_key_iv: "f22f99f4d75f497488d38a10e7ad949b"
server_key_version: "encryptionkey@0"
I20260620 23:09:21.256023 28894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:21.272014 28894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:21.275689 28894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:21.275928 28894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:21.276863 28894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:21.278498 28956 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 23:09:21.283919 28894 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:09:21.284042 28894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 23:09:21.284157 28894 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:09:21.287211 28894 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:09:21.287294 28894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260620 23:09:21.287626 28956 tablet_bootstrap.cc:492] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Bootstrap starting.
I20260620 23:09:21.314450 28956 log.cc:824] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:21.325766 28894 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:38291
I20260620 23:09:21.325789 29063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:38291 every 8 connection(s)
I20260620 23:09:21.327342 28894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:09:21.329349 28956 tablet_bootstrap.cc:492] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:09:21.329954 28956 tablet_bootstrap.cc:492] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Bootstrap complete.
I20260620 23:09:21.330332 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 28894
I20260620 23:09:21.331058 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:46477
--local_ip_for_outbound_sockets=127.12.2.66
--tserver_master_addrs=127.12.2.126:43921
--webserver_port=42581
--webserver_interface=127.12.2.66
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:09:21.331493 28956 ts_tablet_manager.cc:1397] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Time spent bootstrapping tablet: real 0.044s user 0.031s sys 0.011s
I20260620 23:09:21.332834 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:21.330186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:39827)
0620 23:09:21.331102 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:21.331612 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":373,"threads_started":1}
W20260620 23:09:21.333568 29064 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:43921 (0 consecutive failures): Network error: Failed to ping master at 127.12.2.126:43921: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
I20260620 23:09:21.334614 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:21.334177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:36375)
0620 23:09:21.334372 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:21.334465 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 23:09:21.335673 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:21.335288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:50015)
0620 23:09:21.335433 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:21.335523 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:09:21.335948 29064 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:09:21.338640 28956 raft_consensus.cc:348] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:09:21.339519 28956 raft_consensus.cc:729] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f3b986af1940afad0f94e6b5b55756, State: Initialized, Role: FOLLOWER
I20260620 23:09:21.340072 28956 consensus_queue.cc:260] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:09:21.341563 28956 ts_tablet_manager.cc:1428] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 23:09:21.341701 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:21.341280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:44909)
0620 23:09:21.341459 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:21.341537 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 23:09:21.342475 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:21.342180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:41305)
0620 23:09:21.342310 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:21.342374 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 23:09:21.641515 29067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:21.641806 29067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:21.641940 29067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:21.649266 29067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:21.649407 29067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:21.649564 29067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:09:21.659582 29067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:46477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=42581
--tserver_master_addrs=127.12.2.126:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:21.660890 29067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:21.662199 29067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:21.674896 29073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:21.674862 29076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:21.674862 29074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:21.675580 29067 server_base.cc:1034] running on GCE node
I20260620 23:09:21.676173 29067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:21.677086 29067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:21.678298 29067 hybrid_clock.cc:648] HybridClock initialized: now 1781996961678246 us; error 40 us; skew 500 ppm
I20260620 23:09:21.678607 29067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:21 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996961, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:21.681536 29067 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:09:21.682986 29067 webserver.cc:460] Webserver started at http://127.12.2.66:42581/ using document root <none> and password file <none>
I20260620 23:09:21.683640 29067 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:21.683737 29067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:21.688843 29067 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 23:09:21.693097 29083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:21.694519 29067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:09:21.694693 29067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "010fc7adc8ee450197816f182dd0ac24"
format_stamp: "Formatted at 2026-06-20 23:08:36 on dist-test-slave-7g8g"
server_key: "b974840cd6f4c8aeb19f76e651cc7a05"
server_key_iv: "6c4701b5c4ebcc4d109e560f0bec9c62"
server_key_version: "encryptionkey@0"
I20260620 23:09:21.695737 29067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:21.707098 29067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:21.710754 29067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:21.711035 29067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:21.711941 29067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:21.713227 29067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:21.713315 29067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:21.713419 29067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:21.713487 29067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:21.742985 29067 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:46477
I20260620 23:09:21.743072 29195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:46477 every 8 connection(s)
I20260620 23:09:21.744351 29067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:09:21.748801 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:21.747112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:40175)
0620 23:09:21.747784 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:21.747982 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":258,"threads_started":1}
W20260620 23:09:21.749432 29196 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:43921 (0 consecutive failures): Network error: Failed to ping master at 127.12.2.126:43921: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
I20260620 23:09:21.750119 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:21.749841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:40733)
0620 23:09:21.749961 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:21.750024 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 23:09:21.751123 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:21.750647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:46785)
0620 23:09:21.750833 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:21.750984 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 23:09:21.751384 29196 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:09:21.754401 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 29067
I20260620 23:09:21.755079 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:40099
--local_ip_for_outbound_sockets=127.12.2.67
--tserver_master_addrs=127.12.2.126:43921
--webserver_port=46151
--webserver_interface=127.12.2.67
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:22.067633 29199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:22.067924 29199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:22.068101 29199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:22.075798 29199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:22.075914 29199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:22.076079 29199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:09:22.086345 29199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:40099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=46151
--tserver_master_addrs=127.12.2.126:43921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:22.087721 29199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:22.089044 29199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:22.101629 29204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:22.101671 29205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:22.102279 29207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:22.102463 29199 server_base.cc:1034] running on GCE node
I20260620 23:09:22.103096 29199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:22.103981 29199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:22.105180 29199 hybrid_clock.cc:648] HybridClock initialized: now 1781996962105135 us; error 29 us; skew 500 ppm
I20260620 23:09:22.105463 29199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:22 dist-test-slave-7g8g krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996962, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:22.108325 29199 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:09:22.109622 29199 webserver.cc:460] Webserver started at http://127.12.2.67:46151/ using document root <none> and password file <none>
I20260620 23:09:22.110242 29199 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:22.110354 29199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:22.115404 29199 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:09:22.117836 29214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:22.119086 29199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:09:22.119231 29199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "131bc3e501ec4163b3918db0a43ddfb3"
format_stamp: "Formatted at 2026-06-20 23:08:36 on dist-test-slave-7g8g"
server_key: "58c3eaa3568c86f8ab9d53ef8a4b0815"
server_key_iv: "c26f0329923e14fdf7881fb729977af8"
server_key_version: "encryptionkey@0"
I20260620 23:09:22.120124 29199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:22.132237 29199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:22.135816 29199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:22.136052 29199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:22.136932 29199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:22.138204 29199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:22.138306 29199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:22.138408 29199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:22.138475 29199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:22.167657 29199 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:40099
I20260620 23:09:22.167708 29326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:40099 every 8 connection(s)
I20260620 23:09:22.169041 29199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:09:22.173655 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:22.171828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:51315)
0620 23:09:22.172567 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:22.172782 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":310,"threads_started":1}
W20260620 23:09:22.174309 29327 heartbeater.cc:644] Failed to heartbeat to 127.12.2.126:43921 (0 consecutive failures): Network error: Failed to ping master at 127.12.2.126:43921: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
I20260620 23:09:22.175205 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:22.174800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:43697)
0620 23:09:22.174979 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:22.175057 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:09:22.175946 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:22.175662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:48173)
0620 23:09:22.175774 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:22.175844 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 23:09:22.176148 29327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:09:22.176827 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 29199
I20260620 23:09:22.344206 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:22.343367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:54139)
0620 23:09:22.343883 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:22.343979 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":247,"threads_started":1}
I20260620 23:09:22.752571 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:22.751893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:38927)
0620 23:09:22.752286 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:22.752359 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":187,"threads_started":1}
I20260620 23:09:23.116258 29333 raft_consensus.cc:482] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:09:23.116488 29333 raft_consensus.cc:504] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:09:23.117956 29333 leader_election.cc:288] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477), 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099)
Jun 20 23:09:23 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996961, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
Jun 20 23:09:23 dist-test-slave-7g8g krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996961, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:09:23.156208 29337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:23.118630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:34313 (local address 127.12.2.67:40099)
0620 23:09:23.134697 (+ 16067us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:23.134708 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:23.134737 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:09:23.136115 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:23.136124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:23.136445 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:23.136722 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:23.138288 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:23.139448 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:23.140709 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:23.141315 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:23.141473 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:23.149847 (+ 8374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:23.149855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:23.149868 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:23.149906 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:23.152626 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:23.153445 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:23.153451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:23.153457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:23.153610 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:23.154002 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:23.154006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:23.154008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:23.154513 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:23.154674 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:23.155480 (+ 806us) server_negotiation.cc:299] Negotiation successful
0620 23:09:23.155805 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15914,"thread_start_us":18204,"threads_started":1}
I20260620 23:09:23.156265 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:23.118410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:41209 (local address 127.12.2.66:46477)
0620 23:09:23.130314 (+ 11904us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:23.130334 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:23.131601 (+ 1267us) server_negotiation.cc:407] Connection header received
0620 23:09:23.133103 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:23.133111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:23.133496 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:23.133786 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:23.135519 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:23.136694 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:23.138747 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:23.139421 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:23.139584 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:23.149832 (+ 10248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:23.149840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:23.149856 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:23.149898 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:23.152764 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:23.153434 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:23.153442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:23.153448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:23.153590 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:23.154067 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:23.154072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:23.154076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:23.154610 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:23.154783 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:23.155511 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 23:09:23.155885 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11721,"thread_start_us":11560,"threads_started":1}
I20260620 23:09:23.156314 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:23.118236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:46477 (local address 127.12.2.65:41209)
0620 23:09:23.130683 (+ 12447us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:23.130729 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:23.131323 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:23.133975 (+ 2652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:23.133987 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:23.134679 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:23.135333 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:23.135352 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:23.136903 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:23.136908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:23.138617 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:23.138629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:23.139612 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:23.139618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:23.139905 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:23.146861 (+ 6956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:09:23.146899 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:23.149505 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:23.152890 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:23.152893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:23.152896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:23.153294 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:23.153696 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:23.153700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:23.153704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:23.153928 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:23.154739 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:23.154743 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:23.155030 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 23:09:23.155267 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 23:09:23.155835 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12293,"mutex_wait_us":31,"spinlock_wait_cycles":25472,"thread_start_us":12525,"threads_started":1}
I20260620 23:09:23.156375 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:23.118485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:40099 (local address 127.12.2.65:34313)
0620 23:09:23.131114 (+ 12629us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:23.131134 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:23.131343 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:23.136704 (+ 5361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:23.136709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:23.137393 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:23.138112 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:23.138125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:23.139608 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:23.139612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:23.140579 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:23.140590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:23.141487 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:23.141494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:23.141639 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:23.146858 (+ 5219us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:09:23.146920 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:23.149608 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:23.152781 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:23.152789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:23.152804 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:23.153316 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:23.153745 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:23.153749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:23.153752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:23.153897 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:23.154650 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:23.154657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:23.155029 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 23:09:23.155267 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 23:09:23.155978 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12494,"mutex_wait_us":275,"spinlock_wait_cycles":40704,"thread_start_us":11862,"threads_started":1}
W20260620 23:09:23.159613 28951 leader_election.cc:341] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099): Not found: Tablet not found: b6ae3e7532994a7f88fcb300ab8cb8b9
W20260620 23:09:23.159865 28950 leader_election.cc:341] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477): Not found: Tablet not found: b6ae3e7532994a7f88fcb300ab8cb8b9
I20260620 23:09:23.159979 28950 leader_election.cc:302] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3b986af1940afad0f94e6b5b55756; no voters: 010fc7adc8ee450197816f182dd0ac24, 131bc3e501ec4163b3918db0a43ddfb3
I20260620 23:09:23.160363 29333 raft_consensus.cc:2726] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:09:23.177526 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:23.176597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:56199)
0620 23:09:23.177117 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:23.177273 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":242,"threads_started":1}
I20260620 23:09:23.345449 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:23.345027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:49515)
0620 23:09:23.345178 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:23.345296 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 23:09:23.754176 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:23.753394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:44911)
0620 23:09:23.753914 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:23.754014 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":277,"threads_started":1}
I20260620 23:09:24.179251 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:24.178433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:40651)
0620 23:09:24.178969 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:24.179081 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":266,"threads_started":1}
I20260620 23:09:24.347184 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:24.346391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:59155)
0620 23:09:24.346885 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:24.346999 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":206,"threads_started":1}
I20260620 23:09:24.755709 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:24.755019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:34855)
0620 23:09:24.755456 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:24.755537 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":234,"threads_started":1}
I20260620 23:09:25.180716 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:25.180042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:45221)
0620 23:09:25.180470 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:25.180558 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":215,"threads_started":1}
I20260620 23:09:25.244527 29344 raft_consensus.cc:482] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:09:25.244719 29344 raft_consensus.cc:504] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:09:25.245487 29344 leader_election.cc:288] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477), 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099)
W20260620 23:09:25.246428 28951 leader_election.cc:341] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099): Not found: Tablet not found: b6ae3e7532994a7f88fcb300ab8cb8b9
W20260620 23:09:25.246593 28950 leader_election.cc:341] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477): Not found: Tablet not found: b6ae3e7532994a7f88fcb300ab8cb8b9
I20260620 23:09:25.246711 28950 leader_election.cc:302] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3b986af1940afad0f94e6b5b55756; no voters: 010fc7adc8ee450197816f182dd0ac24, 131bc3e501ec4163b3918db0a43ddfb3
I20260620 23:09:25.246958 29344 raft_consensus.cc:2726] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:09:25.348814 29345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:25.348015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:35303)
0620 23:09:25.348488 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:25.348614 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":202,"threads_started":1}
I20260620 23:09:25.757248 29346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:25.756572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:44221)
0620 23:09:25.756998 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:25.757075 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":212,"threads_started":1}
I20260620 23:09:26.182230 29347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:26.181507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:37001)
0620 23:09:26.181967 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:26.182050 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":237,"threads_started":1}
I20260620 23:09:26.350564 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:26.349825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:41993)
0620 23:09:26.350304 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:26.350384 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":253,"threads_started":1}
I20260620 23:09:26.758908 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:26.758189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:56621)
0620 23:09:26.758626 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:26.758725 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":222,"threads_started":1}
I20260620 23:09:27.183830 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:27.183163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:58035)
0620 23:09:27.183579 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:27.183671 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":207,"threads_started":1}
I20260620 23:09:27.351050 29351 raft_consensus.cc:482] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:09:27.351280 29351 raft_consensus.cc:504] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:09:27.351938 29351 leader_election.cc:288] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477), 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099)
I20260620 23:09:27.352216 29352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:27.351397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:39925)
0620 23:09:27.351897 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:27.351997 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":294,"threads_started":1}
W20260620 23:09:27.352792 28951 leader_election.cc:341] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099): Not found: Tablet not found: b6ae3e7532994a7f88fcb300ab8cb8b9
W20260620 23:09:27.352952 28950 leader_election.cc:341] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477): Not found: Tablet not found: b6ae3e7532994a7f88fcb300ab8cb8b9
I20260620 23:09:27.353065 28950 leader_election.cc:302] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3b986af1940afad0f94e6b5b55756; no voters: 010fc7adc8ee450197816f182dd0ac24, 131bc3e501ec4163b3918db0a43ddfb3
I20260620 23:09:27.353282 29351 raft_consensus.cc:2726] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:09:27.760475 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:27.759727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:60219)
0620 23:09:27.760208 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:27.760290 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":258,"threads_started":1}
I20260620 23:09:28.185379 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:28.184655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:37483)
0620 23:09:28.185128 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:28.185209 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260620 23:09:28.353820 29355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:28.353061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:36379)
0620 23:09:28.353534 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:28.353616 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":274,"threads_started":1}
I20260620 23:09:28.762095 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:28.761345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:41639)
0620 23:09:28.761836 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:28.761918 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":195,"threads_started":1}
I20260620 23:09:29.187026 29357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:29.186228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:40153)
0620 23:09:29.186734 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:29.186814 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":214,"threads_started":1}
I20260620 23:09:29.355376 29358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:29.354623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:37181)
0620 23:09:29.355126 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:29.355218 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":238,"threads_started":1}
I20260620 23:09:29.763775 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:29.763041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:56185)
0620 23:09:29.763492 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:29.763578 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":242,"threads_started":1}
I20260620 23:09:30.188537 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:30.187888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:45445)
0620 23:09:30.188273 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:30.188348 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":196,"threads_started":1}
I20260620 23:09:30.356884 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:30.356179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:50049)
0620 23:09:30.356586 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:30.356665 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
I20260620 23:09:30.765348 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:30.764528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:56015)
0620 23:09:30.765086 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:30.765181 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":238,"threads_started":1}
I20260620 23:09:31.190135 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:31.189396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:51201)
0620 23:09:31.189853 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:31.189943 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
I20260620 23:09:31.358605 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:31.357861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:47641)
0620 23:09:31.358307 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:31.358400 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260620 23:09:31.766949 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:31.766210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:49509)
0620 23:09:31.766666 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:31.766753 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":226,"threads_started":1}
I20260620 23:09:32.191795 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:32.191127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:59673)
0620 23:09:32.191535 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:32.191611 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":226,"threads_started":1}
I20260620 23:09:32.360137 29367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:32.359446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:45027)
0620 23:09:32.359857 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:32.359945 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":233,"threads_started":1}
I20260620 23:09:32.395998 29368 raft_consensus.cc:482] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:09:32.396225 29368 raft_consensus.cc:504] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:09:32.396893 29368 leader_election.cc:288] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477), 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099)
W20260620 23:09:32.397825 28950 leader_election.cc:341] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477): Not found: Tablet not found: b6ae3e7532994a7f88fcb300ab8cb8b9
W20260620 23:09:32.398151 28951 leader_election.cc:341] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099): Not found: Tablet not found: b6ae3e7532994a7f88fcb300ab8cb8b9
I20260620 23:09:32.398329 28951 leader_election.cc:302] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3b986af1940afad0f94e6b5b55756; no voters: 010fc7adc8ee450197816f182dd0ac24, 131bc3e501ec4163b3918db0a43ddfb3
I20260620 23:09:32.398635 29368 raft_consensus.cc:2726] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:09:32.768435 29369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:32.767764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:35695)
0620 23:09:32.768189 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:32.768274 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":217,"threads_started":1}
I20260620 23:09:33.193311 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:33.192617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:53157)
0620 23:09:33.193056 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:33.193139 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":212,"threads_started":1}
I20260620 23:09:33.361707 29371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:33.360991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:46843)
0620 23:09:33.361445 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:33.361535 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":224,"threads_started":1}
I20260620 23:09:33.769912 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:33.769220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:48013)
0620 23:09:33.769664 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:33.769743 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":205,"threads_started":1}
I20260620 23:09:34.194967 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:34.194208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:52319)
0620 23:09:34.194676 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:34.194762 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":227,"threads_started":1}
I20260620 23:09:34.363425 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:34.362607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:47399)
0620 23:09:34.363112 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:34.363213 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":244,"threads_started":1}
I20260620 23:09:34.771739 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:34.770772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:53825)
0620 23:09:34.771326 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:34.771490 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":304,"threads_started":1}
I20260620 23:09:35.196534 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:35.195810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:47247)
0620 23:09:35.196275 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:35.196372 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":230,"threads_started":1}
I20260620 23:09:35.365161 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:35.364460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:39051)
0620 23:09:35.364892 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:35.364979 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":231,"threads_started":1}
I20260620 23:09:35.773365 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:35.772648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:51821)
0620 23:09:35.773106 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:35.773189 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":216,"threads_started":1}
I20260620 23:09:36.198108 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:36.197402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:47355)
0620 23:09:36.197850 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:36.197926 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":257,"threads_started":1}
I20260620 23:09:36.367259 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:36.366303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.65:47325)
0620 23:09:36.366813 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:36.366917 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":258,"threads_started":1}
I20260620 23:09:36.775126 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:36.774380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.66:60507)
0620 23:09:36.774807 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:36.774885 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":198,"threads_started":1}
I20260620 23:09:37.058393 29382 raft_consensus.cc:482] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:09:37.058633 29382 raft_consensus.cc:504] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010fc7adc8ee450197816f182dd0ac24" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 46477 } } peers { permanent_uuid: "27f3b986af1940afad0f94e6b5b55756" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 38291 } } peers { permanent_uuid: "131bc3e501ec4163b3918db0a43ddfb3" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40099 } }
I20260620 23:09:37.059307 29382 leader_election.cc:288] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477), 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099)
W20260620 23:09:37.060317 28951 leader_election.cc:341] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 131bc3e501ec4163b3918db0a43ddfb3 (127.12.2.67:40099): Not found: Tablet not found: b6ae3e7532994a7f88fcb300ab8cb8b9
W20260620 23:09:37.060485 28950 leader_election.cc:341] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 010fc7adc8ee450197816f182dd0ac24 (127.12.2.66:46477): Not found: Tablet not found: b6ae3e7532994a7f88fcb300ab8cb8b9
I20260620 23:09:37.060649 28950 leader_election.cc:302] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3b986af1940afad0f94e6b5b55756; no voters: 010fc7adc8ee450197816f182dd0ac24, 131bc3e501ec4163b3918db0a43ddfb3
I20260620 23:09:37.060923 29382 raft_consensus.cc:2726] T b6ae3e7532994a7f88fcb300ab8cb8b9 P 27f3b986af1940afad0f94e6b5b55756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:09:37.181685 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 28894
I20260620 23:09:37.199820 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:37.182747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.12.2.65:38291)
0620 23:09:37.183394 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:37.183404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:37.183436 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:09:37.183521 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:37.183530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:37.183835 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:37.184001 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:37.185421 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:37.186399 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:37.187573 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:37.188134 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:37.188231 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:37.194064 (+ 5833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:37.194078 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:37.194095 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:37.194133 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:37.196949 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:37.197620 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:37.197628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:37.197631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:37.197776 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:37.198078 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:37.198081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:37.198083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:37.198491 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:37.198806 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:37.199224 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 23:09:37.199492 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":288,"threads_started":1}
I20260620 23:09:37.200134 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:37.199252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43921 (local address 127.12.2.67:50329)
0620 23:09:37.199821 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:37.199928 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.126:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":391,"threads_started":1}
I20260620 23:09:37.201079 29059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:37.289448 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 28894
I20260620 23:09:37.305702 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 29067
I20260620 23:09:37.323446 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:37.306405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.12.2.66:46477)
0620 23:09:37.307003 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:37.307009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:37.307025 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:09:37.307079 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:37.307083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:37.307148 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:37.307238 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:37.308519 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:37.309288 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:37.310450 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:37.310890 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:37.311044 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:37.318307 (+ 7263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:37.318312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:37.318315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:37.318351 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:37.320747 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:37.321378 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:37.321382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:37.321385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:37.321510 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:37.321883 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:37.321888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:37.321891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:37.322172 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:37.322400 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:37.322701 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 23:09:37.322914 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":322,"threads_started":1}
I20260620 23:09:37.324158 29191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:37.417112 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 29067
I20260620 23:09:37.431348 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 29199
I20260620 23:09:37.444118 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:37.432175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.12.2.67:40099)
0620 23:09:37.432777 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:37.432782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:37.432799 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:09:37.432852 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:37.432856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:37.432923 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:37.433013 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:37.434299 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:37.435105 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:37.436279 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:37.436719 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:37.436812 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:37.439242 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:37.439247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:37.439250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:37.439283 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:37.441826 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:37.442458 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:37.442461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:37.442463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:37.442603 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:37.442863 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:37.442866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:37.442868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:37.443149 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:37.443356 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:37.443581 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 23:09:37.443748 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":342,"threads_started":1}
I20260620 23:09:37.444869 29322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:37.525872 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 29199
2026-06-20T23:09:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63897 ms)
[----------] 30 tests from SecurityITest (280888 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:09:37 dist-test-slave-7g8g krb5kdc[29392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:09:37 dist-test-slave-7g8g krb5kdc[29392](info): set up 2 sockets
Jun 20 23:09:37 dist-test-slave-7g8g krb5kdc[29392](info): commencing operation
krb5kdc: starting...
W20260620 23:09:38.530684 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:09:38 dist-test-slave-7g8g krb5kdc[29392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:09:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:09:38.756079 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:35551
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:35551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:39.068421 29405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:39.068707 29405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:39.068840 29405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:39.076148 29405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:09:39.076259 29405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:39.076332 29405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:39.076402 29405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:09:39.076460 29405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:09:39.086341 29405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:35551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:35551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:39.087667 29405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:39.088920 29405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:39.100076 29410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:39.100116 29411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:39.100145 29413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:39.100522 29405 server_base.cc:1034] running on GCE node
I20260620 23:09:39.101467 29405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:39.102864 29405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:39.104081 29405 hybrid_clock.cc:648] HybridClock initialized: now 1781996979104032 us; error 34 us; skew 500 ppm
I20260620 23:09:39.104421 29405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:39 dist-test-slave-7g8g krb5kdc[29392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996979, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:39.107470 29405 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:09:39.108839 29405 webserver.cc:460] Webserver started at http://127.12.2.126:36833/ using document root <none> and password file <none>
I20260620 23:09:39.109495 29405 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:39.109601 29405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:39.109875 29405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:39.112778 29405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "7dfac99fe5ae4b2cb42f08674149f077"
format_stamp: "Formatted at 2026-06-20 23:09:39 on dist-test-slave-7g8g"
server_key: "228b17db2b574cc60b0390cb4bb7dc1a"
server_key_iv: "7566387d3c4fcdb6072729e7060b9bc2"
server_key_version: "encryptionkey@0"
I20260620 23:09:39.113499 29405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "7dfac99fe5ae4b2cb42f08674149f077"
format_stamp: "Formatted at 2026-06-20 23:09:39 on dist-test-slave-7g8g"
server_key: "228b17db2b574cc60b0390cb4bb7dc1a"
server_key_iv: "7566387d3c4fcdb6072729e7060b9bc2"
server_key_version: "encryptionkey@0"
I20260620 23:09:39.118461 29405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:09:39.121759 29420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:39.123083 29405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:09:39.123243 29405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "7dfac99fe5ae4b2cb42f08674149f077"
format_stamp: "Formatted at 2026-06-20 23:09:39 on dist-test-slave-7g8g"
server_key: "228b17db2b574cc60b0390cb4bb7dc1a"
server_key_iv: "7566387d3c4fcdb6072729e7060b9bc2"
server_key_version: "encryptionkey@0"
I20260620 23:09:39.123417 29405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:39.143749 29405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:39.147578 29405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:39.147817 29405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:39.164589 29405 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:35551
I20260620 23:09:39.164624 29471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:35551 every 8 connection(s)
I20260620 23:09:39.165863 29405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:09:39.167455 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 29405
I20260620 23:09:39.167651 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:09:39.168248 12297 external_mini_cluster.cc:1428] Setting key 08a13df1017d66ec2129bae1619df630
I20260620 23:09:39.169902 29472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:09:39 dist-test-slave-7g8g krb5kdc[29392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:39.183475 29472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077: Bootstrap starting.
I20260620 23:09:39.187534 29472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:39.188752 29472 log.cc:824] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:39.192057 29472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077: No bootstrap required, opened a new log
I20260620 23:09:39.192498 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:39.170067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.12.2.126:35551)
0620 23:09:39.170806 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:39.170821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:39.170873 (+ 52us) server_negotiation.cc:407] Connection header received
0620 23:09:39.173084 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:39.173113 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:39.173656 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:39.174109 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:39.175487 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:39.177500 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:39.178782 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:39.179843 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:39.180035 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:39.184120 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:39.184131 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:39.184149 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:39.184199 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:39.187998 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:39.188683 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:39.188691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:39.188700 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:39.188943 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:39.189339 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:39.189346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:39.189349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:39.190105 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:39.190376 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:39.190750 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 23:09:39.191197 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":509,"thread_start_us":307,"threads_started":1}
I20260620 23:09:39.199534 29472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfac99fe5ae4b2cb42f08674149f077" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35551 } }
I20260620 23:09:39.199880 29472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:39.199960 29472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dfac99fe5ae4b2cb42f08674149f077, State: Initialized, Role: FOLLOWER
I20260620 23:09:39.200433 29472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfac99fe5ae4b2cb42f08674149f077" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35551 } }
I20260620 23:09:39.200620 29472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:09:39.200713 29472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:09:39.200841 29472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:39.202967 29472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfac99fe5ae4b2cb42f08674149f077" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35551 } }
I20260620 23:09:39.203408 29472 leader_election.cc:302] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dfac99fe5ae4b2cb42f08674149f077; no voters:
I20260620 23:09:39.203900 29472 leader_election.cc:288] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:09:39.204031 29477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:39.204593 29477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [term 1 LEADER]: Becoming Leader. State: Replica: 7dfac99fe5ae4b2cb42f08674149f077, State: Running, Role: LEADER
I20260620 23:09:39.205098 29477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfac99fe5ae4b2cb42f08674149f077" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35551 } }
I20260620 23:09:39.205787 29472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:09:39.207644 29478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dfac99fe5ae4b2cb42f08674149f077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfac99fe5ae4b2cb42f08674149f077" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35551 } } }
I20260620 23:09:39.208004 29478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:39.208747 29479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dfac99fe5ae4b2cb42f08674149f077. Latest consensus state: current_term: 1 leader_uuid: "7dfac99fe5ae4b2cb42f08674149f077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dfac99fe5ae4b2cb42f08674149f077" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35551 } } }
I20260620 23:09:39.209115 29479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077 [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:39.209506 29485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:09:39.213994 29485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:09:39.222260 29485 catalog_manager.cc:1347] Generated new cluster ID: 71c3320bd562461dbf2ca3c4c256f56e
I20260620 23:09:39.222368 29485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:09:39.245579 29485 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:09:39.247411 29485 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:09:39.257771 29485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077: Generated new TSK 0
I20260620 23:09:39.258561 29485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:09:39.312479 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:35551
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:39.630903 29498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:39.631188 29498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:39.631332 29498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:39.638342 29498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:39.638444 29498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:39.638589 29498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:09:39.648005 29498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:35551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:39.649242 29498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:39.650516 29498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:39.666870 29506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:39.666898 29504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:39.666841 29503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:39.667632 29498 server_base.cc:1034] running on GCE node
I20260620 23:09:39.668174 29498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:39.669024 29498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:39.670219 29498 hybrid_clock.cc:648] HybridClock initialized: now 1781996979670177 us; error 35 us; skew 500 ppm
I20260620 23:09:39.670518 29498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:39 dist-test-slave-7g8g krb5kdc[29392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996979, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:39.673484 29498 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:09:39.674857 29498 webserver.cc:460] Webserver started at http://127.12.2.65:44441/ using document root <none> and password file <none>
I20260620 23:09:39.675515 29498 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:39.675634 29498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:39.675913 29498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:39.678782 29498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "019b469185284934b62c0529c969aba9"
format_stamp: "Formatted at 2026-06-20 23:09:39 on dist-test-slave-7g8g"
server_key: "3490e952e4ed370438e7c8e2f88aa671"
server_key_iv: "6e921d5427aea36cd1701ecaefdc2ecd"
server_key_version: "encryptionkey@0"
I20260620 23:09:39.679553 29498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "019b469185284934b62c0529c969aba9"
format_stamp: "Formatted at 2026-06-20 23:09:39 on dist-test-slave-7g8g"
server_key: "3490e952e4ed370438e7c8e2f88aa671"
server_key_iv: "6e921d5427aea36cd1701ecaefdc2ecd"
server_key_version: "encryptionkey@0"
I20260620 23:09:39.684449 29498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:09:39.687865 29513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:39.689137 29498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:09:39.689285 29498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "019b469185284934b62c0529c969aba9"
format_stamp: "Formatted at 2026-06-20 23:09:39 on dist-test-slave-7g8g"
server_key: "3490e952e4ed370438e7c8e2f88aa671"
server_key_iv: "6e921d5427aea36cd1701ecaefdc2ecd"
server_key_version: "encryptionkey@0"
I20260620 23:09:39.689468 29498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:39.702483 29498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:39.706321 29498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:39.706580 29498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:39.707602 29498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:39.709014 29498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:39.709090 29498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:39.709200 29498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:39.709290 29498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 23:09:39.738659 29498 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:37261
I20260620 23:09:39.738706 29625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:37261 every 8 connection(s)
I20260620 23:09:39.739933 29498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:09:39.748118 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 29498
I20260620 23:09:39.748332 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:09:39.748965 12297 external_mini_cluster.cc:1428] Setting key 1ebac378cec71d2e12cde2c8d2a08c5b
Jun 20 23:09:39 dist-test-slave-7g8g krb5kdc[29392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996979, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:39.759196 29628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:39.742394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:59511 (local address 127.12.2.126:35551)
0620 23:09:39.742855 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:39.742862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:39.743809 (+ 947us) server_negotiation.cc:407] Connection header received
0620 23:09:39.745530 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:39.745535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:39.745600 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:39.745693 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:39.747123 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:39.748018 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:39.749578 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:39.750107 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:39.750225 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:39.753744 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:39.753749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:39.753753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:39.753786 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:39.756060 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:39.756760 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:39.756764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:39.756766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:39.756890 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:39.757324 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:39.757330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:39.757333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:39.757692 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:39.757853 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:39.758551 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 23:09:39.758772 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":215,"threads_started":1}
I20260620 23:09:39.760171 29629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:39.742965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:35551 (local address 127.12.2.65:59511)
0620 23:09:39.743651 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:39.743692 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:39.745272 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:39.745873 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:39.745880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:39.746388 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:39.746922 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:39.746969 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:39.748158 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:39.748161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:39.749441 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:39.749453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:39.750323 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:39.750332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:39.750588 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:39.751271 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:09:39.751293 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:39.753528 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:39.756194 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:39.756204 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:39.756223 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:39.756633 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:39.756993 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:39.756998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:39.757002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:39.757190 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:39.757807 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:39.757814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:39.758196 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 23:09:39.758414 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 23:09:39.758807 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":260,"threads_started":1}
I20260620 23:09:39.761772 29626 heartbeater.cc:347] Connected to a master server at 127.12.2.126:35551
I20260620 23:09:39.762095 29626 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:39.762877 29626 heartbeater.cc:511] Master 127.12.2.126:35551 requested a full tablet report, sending...
I20260620 23:09:39.764951 29437 ts_manager.cc:194] Registered new tserver with Master: 019b469185284934b62c0529c969aba9 (127.12.2.65:37261)
I20260620 23:09:39.766403 29437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:59511
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:09:39.799719 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:35551
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:40.114050 29632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:40.114332 29632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:40.114454 29632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:40.121569 29632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:40.121677 29632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:40.121825 29632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:09:40.131451 29632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:35551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:40.132676 29632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:40.133927 29632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:40.146159 29640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:40.146225 29637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:40.146190 29638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:40.147228 29632 server_base.cc:1034] running on GCE node
I20260620 23:09:40.147785 29632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:40.148635 29632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:40.149819 29632 hybrid_clock.cc:648] HybridClock initialized: now 1781996980149770 us; error 41 us; skew 500 ppm
I20260620 23:09:40.150095 29632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:40 dist-test-slave-7g8g krb5kdc[29392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:40.153083 29632 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:09:40.154465 29632 webserver.cc:460] Webserver started at http://127.12.2.66:38987/ using document root <none> and password file <none>
I20260620 23:09:40.155196 29632 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:40.155308 29632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:40.155581 29632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:40.158308 29632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "19db49c648104c8da42af619064b7c61"
format_stamp: "Formatted at 2026-06-20 23:09:40 on dist-test-slave-7g8g"
server_key: "2a1519537cf633e9eb16abb31ca8f936"
server_key_iv: "a18bad285097ed4bac76236dc17a66cf"
server_key_version: "encryptionkey@0"
I20260620 23:09:40.159024 29632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "19db49c648104c8da42af619064b7c61"
format_stamp: "Formatted at 2026-06-20 23:09:40 on dist-test-slave-7g8g"
server_key: "2a1519537cf633e9eb16abb31ca8f936"
server_key_iv: "a18bad285097ed4bac76236dc17a66cf"
server_key_version: "encryptionkey@0"
I20260620 23:09:40.163756 29632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:09:40.167066 29647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:40.168357 29632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:09:40.168532 29632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "19db49c648104c8da42af619064b7c61"
format_stamp: "Formatted at 2026-06-20 23:09:40 on dist-test-slave-7g8g"
server_key: "2a1519537cf633e9eb16abb31ca8f936"
server_key_iv: "a18bad285097ed4bac76236dc17a66cf"
server_key_version: "encryptionkey@0"
I20260620 23:09:40.168795 29632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:40.181737 29632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:40.185595 29632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:40.185849 29632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:40.186766 29632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:40.188252 29632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:40.188341 29632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:40.188444 29632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:40.188489 29632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:40.218214 29632 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:35575
I20260620 23:09:40.218259 29759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:35575 every 8 connection(s)
I20260620 23:09:40.219451 29632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:09:40.221331 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 29632
I20260620 23:09:40.221498 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:09:40.221927 12297 external_mini_cluster.cc:1428] Setting key 003f337956dc19c3c13c81993682d31c
Jun 20 23:09:40 dist-test-slave-7g8g krb5kdc[29392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:40.239087 29628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.222044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:36251 (local address 127.12.2.126:35551)
0620 23:09:40.222236 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:40.222242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:40.223547 (+ 1305us) server_negotiation.cc:407] Connection header received
0620 23:09:40.225388 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:40.225393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:40.225486 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:40.225570 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:40.227164 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.228004 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.229508 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.229910 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.230019 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.233592 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:40.233597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:40.233601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:40.233634 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:40.236127 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:40.236856 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:40.236860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:40.236863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:40.236994 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:40.237401 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:40.237406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:40.237408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:40.237667 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:40.237807 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:40.238487 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 23:09:40.238717 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:09:40.240073 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.222558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:35551 (local address 127.12.2.66:36251)
0620 23:09:40.223396 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:40.223449 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:40.225130 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:40.225818 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:40.225826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:40.226390 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:40.227012 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.227030 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.228151 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.228156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.229372 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.229384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.230117 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.230124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.230347 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.231069 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:09:40.231099 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:40.233377 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:40.236271 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:40.236281 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:40.236298 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:40.236735 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:40.237097 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:40.237102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:40.237106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:40.237294 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:40.237786 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:40.237795 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:40.238138 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 23:09:40.238365 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 23:09:40.238907 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":364,"threads_started":1}
I20260620 23:09:40.241515 29760 heartbeater.cc:347] Connected to a master server at 127.12.2.126:35551
I20260620 23:09:40.241801 29760 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:40.242506 29760 heartbeater.cc:511] Master 127.12.2.126:35551 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
I20260620 23:09:40.244000 29437 ts_manager.cc:194] Registered new tserver with Master: 19db49c648104c8da42af619064b7c61 (127.12.2.66:35575)
I20260620 23:09:40.244787 29437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:36251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:09:40.270960 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:35551
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:40.577242 29765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:40.577499 29765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:40.577630 29765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:40.584730 29765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:40.584846 29765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:40.584998 29765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:09:40.594631 29765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:35551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:40.595855 29765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:40.597126 29765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:40.609058 29770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:40.608989 29771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:40.609020 29773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:40.609958 29765 server_base.cc:1034] running on GCE node
I20260620 23:09:40.610548 29765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:40.611476 29765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:40.612654 29765 hybrid_clock.cc:648] HybridClock initialized: now 1781996980612599 us; error 43 us; skew 500 ppm
I20260620 23:09:40.612926 29765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:40 dist-test-slave-7g8g krb5kdc[29392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:40.615769 29765 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:09:40.617057 29765 webserver.cc:460] Webserver started at http://127.12.2.67:36213/ using document root <none> and password file <none>
I20260620 23:09:40.617695 29765 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:40.617813 29765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:40.618116 29765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:40.620851 29765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "d05569a1c78a4c8ca163670b3ec9e184"
format_stamp: "Formatted at 2026-06-20 23:09:40 on dist-test-slave-7g8g"
server_key: "293eb7cdcfd35f26a830dd5acbed3ade"
server_key_iv: "105041115947736d321f2785da33f870"
server_key_version: "encryptionkey@0"
I20260620 23:09:40.621590 29765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "d05569a1c78a4c8ca163670b3ec9e184"
format_stamp: "Formatted at 2026-06-20 23:09:40 on dist-test-slave-7g8g"
server_key: "293eb7cdcfd35f26a830dd5acbed3ade"
server_key_iv: "105041115947736d321f2785da33f870"
server_key_version: "encryptionkey@0"
I20260620 23:09:40.626422 29765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260620 23:09:40.629740 29780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:40.631007 29765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:09:40.631151 29765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "d05569a1c78a4c8ca163670b3ec9e184"
format_stamp: "Formatted at 2026-06-20 23:09:40 on dist-test-slave-7g8g"
server_key: "293eb7cdcfd35f26a830dd5acbed3ade"
server_key_iv: "105041115947736d321f2785da33f870"
server_key_version: "encryptionkey@0"
I20260620 23:09:40.631357 29765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:40.652719 29765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:40.656628 29765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:40.656901 29765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:40.657840 29765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:40.659224 29765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:40.659302 29765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:40.659416 29765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:40.659464 29765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:40.688694 29765 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:46605
I20260620 23:09:40.688761 29892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:46605 every 8 connection(s)
I20260620 23:09:40.689898 29765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:09:40.692575 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 29765
I20260620 23:09:40.692785 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:09:40.693351 12297 external_mini_cluster.cc:1428] Setting key 03149de7e5f9750c821af770e1c710f4
Jun 20 23:09:40 dist-test-slave-7g8g krb5kdc[29392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:40.709254 29628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.692327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:54797 (local address 127.12.2.126:35551)
0620 23:09:40.692514 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:40.692519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:40.694365 (+ 1846us) server_negotiation.cc:407] Connection header received
0620 23:09:40.696086 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:40.696090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:40.696162 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:40.696247 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:40.697679 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.698383 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.699882 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.700279 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.700367 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.703749 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:40.703754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:40.703758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:40.703789 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:40.706111 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:40.706890 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:40.706894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:40.706896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:40.707047 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:40.707573 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:40.707577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:40.707580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:40.707916 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:40.708058 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:40.708836 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 23:09:40.709003 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:09:40.710274 29895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.692963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:35551 (local address 127.12.2.67:54797)
0620 23:09:40.693919 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:40.694018 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:40.695818 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:40.696444 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:40.696454 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:40.696978 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:40.697546 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.697563 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.698515 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.698519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.699776 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.699789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.700484 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.700489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.700684 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.701323 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:09:40.701344 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:40.703550 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:40.706243 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:40.706250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:40.706263 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:40.706764 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:40.707204 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:40.707209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:40.707212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:40.707460 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:40.708033 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:40.708039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:40.708440 (+ 401us) client_negotiation.cc:769] Sending connection context
0620 23:09:40.708735 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 23:09:40.709152 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":333,"threads_started":1}
I20260620 23:09:40.711562 29893 heartbeater.cc:347] Connected to a master server at 127.12.2.126:35551
I20260620 23:09:40.711828 29893 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:40.712472 29893 heartbeater.cc:511] Master 127.12.2.126:35551 requested a full tablet report, sending...
I20260620 23:09:40.713752 29437 ts_manager.cc:194] Registered new tserver with Master: d05569a1c78a4c8ca163670b3ec9e184 (127.12.2.67:46605)
I20260620 23:09:40.714546 29437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:54797
I20260620 23:09:40.720566 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:09:40.736294 29628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.724794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.12.2.126:35551)
0620 23:09:40.724997 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:40.725003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:40.725334 (+ 331us) server_negotiation.cc:407] Connection header received
0620 23:09:40.725488 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:40.725491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:40.725552 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:40.725623 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:40.727041 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.727764 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.729005 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.729380 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.729469 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.731608 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:40.731614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:40.731617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:40.731648 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:40.734034 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:40.734648 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:40.734652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:40.734654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:40.734774 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:40.735068 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:40.735072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:40.735076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:40.735452 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:40.735627 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:40.735897 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:09:40.736079 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:09:40.739895 29437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:09:40.746490 29437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:09:40.769838 29626 heartbeater.cc:503] Master 127.12.2.126:35551 was elected leader, sending a full tablet report...
I20260620 23:09:40.777033 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.757714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.12.2.66:35575)
0620 23:09:40.758408 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:40.758416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:40.768003 (+ 9587us) server_negotiation.cc:407] Connection header received
0620 23:09:40.768419 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:40.768429 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:40.768724 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:40.768889 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:40.769900 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.771437 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.774433 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.775592 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.775724 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.775917 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:40.776583 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 23:09:40.776776 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":1220,"threads_started":1}
I20260620 23:09:40.777823 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.757590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:35575 (local address 127.0.0.1:60922)
0620 23:09:40.767848 (+ 10258us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:40.767891 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:40.768164 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:40.769227 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:40.769235 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:40.769268 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:40.769703 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.769715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.771677 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.771682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.774289 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.774305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.776061 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.776067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.776263 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.776355 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 23:09:40.777270 (+ 915us) client_negotiation.cc:240] Negotiation successful
0620 23:09:40.777500 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10020,"mutex_wait_us":91,"thread_start_us":15852,"threads_started":1}
I20260620 23:09:40.781121 29910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.758669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.12.2.67:46605)
0620 23:09:40.767524 (+ 8855us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:40.767534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:40.773154 (+ 5620us) server_negotiation.cc:407] Connection header received
0620 23:09:40.773303 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:40.773311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:40.773517 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:40.773686 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:40.774470 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.776139 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.778799 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.780124 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.780236 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.780442 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:40.780654 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 23:09:40.780806 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8659,"thread_start_us":8486,"threads_started":1}
I20260620 23:09:40.781127 29907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.758123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:46605 (local address 127.0.0.1:55230)
0620 23:09:40.772875 (+ 14752us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:40.772891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:40.773018 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:40.773680 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:40.773684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:40.773726 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:40.774299 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.774308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.776269 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.776278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.778610 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.778624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.780298 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.780304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.780472 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.780505 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:09:40.780605 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:09:40.780728 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14608,"mutex_wait_us":155,"spinlock_wait_cycles":9344,"thread_start_us":9365,"threads_started":1}
I20260620 23:09:40.781602 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.757806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:37261 (local address 127.0.0.1:35152)
0620 23:09:40.772472 (+ 14666us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:40.772495 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:40.772628 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:40.773677 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:40.773682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:40.773726 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:40.774241 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.774256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.776220 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.776225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.778613 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.778624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.780601 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.780607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.780758 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.780786 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:09:40.780871 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:09:40.781003 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14523,"mutex_wait_us":98,"spinlock_wait_cycles":5760,"thread_start_us":14788,"threads_started":1}
I20260620 23:09:40.781654 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.758611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.12.2.65:37261)
0620 23:09:40.771185 (+ 12574us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:40.771194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:40.772750 (+ 1556us) server_negotiation.cc:407] Connection header received
0620 23:09:40.772840 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:40.772848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:40.773085 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:40.773275 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:40.774469 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.776056 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.778770 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.780147 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.780264 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.780472 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:40.781215 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 23:09:40.781372 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12393,"thread_start_us":12210,"threads_started":1}
I20260620 23:09:40.781836 29695 tablet_service.cc:1467] Processing CreateTablet for tablet 2e11d7cb61704bc58625432d0eb48941 (DEFAULT_TABLE table=test-table [id=392f7ca62e1b430e880b77b7a9821d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:40.783221 29695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e11d7cb61704bc58625432d0eb48941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:40.784842 29561 tablet_service.cc:1467] Processing CreateTablet for tablet 2e11d7cb61704bc58625432d0eb48941 (DEFAULT_TABLE table=test-table [id=392f7ca62e1b430e880b77b7a9821d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:40.784842 29828 tablet_service.cc:1467] Processing CreateTablet for tablet 2e11d7cb61704bc58625432d0eb48941 (DEFAULT_TABLE table=test-table [id=392f7ca62e1b430e880b77b7a9821d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:40.786157 29561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e11d7cb61704bc58625432d0eb48941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:40.786170 29828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e11d7cb61704bc58625432d0eb48941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:40.795203 29911 tablet_bootstrap.cc:492] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: Bootstrap starting.
I20260620 23:09:40.797866 29911 tablet_bootstrap.cc:654] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:40.798806 29911 log.cc:824] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:40.800113 29913 tablet_bootstrap.cc:492] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: Bootstrap starting.
I20260620 23:09:40.801278 29914 tablet_bootstrap.cc:492] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: Bootstrap starting.
I20260620 23:09:40.801755 29911 tablet_bootstrap.cc:492] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: No bootstrap required, opened a new log
I20260620 23:09:40.801999 29911 ts_tablet_manager.cc:1397] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 23:09:40.804527 29913 tablet_bootstrap.cc:654] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:40.804840 29914 tablet_bootstrap.cc:654] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:40.805478 29913 log.cc:824] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:40.806038 29914 log.cc:824] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:40.808054 29913 tablet_bootstrap.cc:492] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: No bootstrap required, opened a new log
I20260620 23:09:40.808284 29913 ts_tablet_manager.cc:1397] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 23:09:40.812626 29911 raft_consensus.cc:348] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19db49c648104c8da42af619064b7c61" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 35575 } } peers { permanent_uuid: "019b469185284934b62c0529c969aba9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37261 } } peers { permanent_uuid: "d05569a1c78a4c8ca163670b3ec9e184" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 46605 } }
I20260620 23:09:40.813091 29911 raft_consensus.cc:374] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:40.813194 29911 raft_consensus.cc:729] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19db49c648104c8da42af619064b7c61, State: Initialized, Role: FOLLOWER
I20260620 23:09:40.813835 29911 consensus_queue.cc:260] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19db49c648104c8da42af619064b7c61" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 35575 } } peers { permanent_uuid: "019b469185284934b62c0529c969aba9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37261 } } peers { permanent_uuid: "d05569a1c78a4c8ca163670b3ec9e184" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 46605 } }
I20260620 23:09:40.814993 29760 heartbeater.cc:503] Master 127.12.2.126:35551 was elected leader, sending a full tablet report...
I20260620 23:09:40.815623 29911 ts_tablet_manager.cc:1428] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 23:09:40.816774 29913 raft_consensus.cc:348] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19db49c648104c8da42af619064b7c61" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 35575 } } peers { permanent_uuid: "019b469185284934b62c0529c969aba9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37261 } } peers { permanent_uuid: "d05569a1c78a4c8ca163670b3ec9e184" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 46605 } }
I20260620 23:09:40.817255 29913 raft_consensus.cc:374] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:40.817353 29913 raft_consensus.cc:729] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05569a1c78a4c8ca163670b3ec9e184, State: Initialized, Role: FOLLOWER
I20260620 23:09:40.817909 29913 consensus_queue.cc:260] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19db49c648104c8da42af619064b7c61" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 35575 } } peers { permanent_uuid: "019b469185284934b62c0529c969aba9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37261 } } peers { permanent_uuid: "d05569a1c78a4c8ca163670b3ec9e184" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 46605 } }
I20260620 23:09:40.818462 29914 tablet_bootstrap.cc:492] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: No bootstrap required, opened a new log
I20260620 23:09:40.818770 29914 ts_tablet_manager.cc:1397] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20260620 23:09:40.818965 29893 heartbeater.cc:503] Master 127.12.2.126:35551 was elected leader, sending a full tablet report...
I20260620 23:09:40.819578 29913 ts_tablet_manager.cc:1428] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 23:09:40.826906 29914 raft_consensus.cc:348] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19db49c648104c8da42af619064b7c61" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 35575 } } peers { permanent_uuid: "019b469185284934b62c0529c969aba9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37261 } } peers { permanent_uuid: "d05569a1c78a4c8ca163670b3ec9e184" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 46605 } }
I20260620 23:09:40.827302 29914 raft_consensus.cc:374] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:40.827378 29914 raft_consensus.cc:729] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019b469185284934b62c0529c969aba9, State: Initialized, Role: FOLLOWER
I20260620 23:09:40.827905 29914 consensus_queue.cc:260] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19db49c648104c8da42af619064b7c61" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 35575 } } peers { permanent_uuid: "019b469185284934b62c0529c969aba9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37261 } } peers { permanent_uuid: "d05569a1c78a4c8ca163670b3ec9e184" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 46605 } }
I20260620 23:09:40.829357 29914 ts_tablet_manager.cc:1428] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
W20260620 23:09:40.941433 29894 tablet.cc:2369] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:09:40.971271 29761 tablet.cc:2369] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:09:40.987306 29918 raft_consensus.cc:482] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:09:40.987677 29918 raft_consensus.cc:504] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19db49c648104c8da42af619064b7c61" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 35575 } } peers { permanent_uuid: "019b469185284934b62c0529c969aba9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37261 } } peers { permanent_uuid: "d05569a1c78a4c8ca163670b3ec9e184" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 46605 } }
I20260620 23:09:40.989848 29918 leader_election.cc:288] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19db49c648104c8da42af619064b7c61 (127.12.2.66:35575), 019b469185284934b62c0529c969aba9 (127.12.2.65:37261)
W20260620 23:09:40.992199 29627 tablet.cc:2369] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:09:40.996368 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.990268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:42219 (local address 127.12.2.66:35575)
0620 23:09:40.990472 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:40.990478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:40.990700 (+ 222us) server_negotiation.cc:407] Connection header received
0620 23:09:40.990970 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:40.990976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:40.991052 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:40.991132 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:40.991538 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.992448 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.994360 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.995555 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.995655 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.995729 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:40.996055 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 23:09:40.996172 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:09:40.996397 29895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.990266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:35575 (local address 127.12.2.67:42219)
0620 23:09:40.990600 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:40.990617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:40.990747 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:40.991179 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:40.991185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:40.991212 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:40.991394 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.991402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.992571 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.992595 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.994216 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.994228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.995691 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.995697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.995877 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.995912 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 23:09:40.996004 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:09:40.996133 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260620 23:09:40.997424 29920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.990696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:37261 (local address 127.12.2.67:49617)
0620 23:09:40.991423 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:40.991437 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:40.991556 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:40.991834 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:40.991838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:40.991863 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:40.992173 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.992181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.993561 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.993565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.995382 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:40.995398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.996834 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.996840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:40.997001 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.997034 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:09:40.997111 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 23:09:40.997200 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":391,"threads_started":1}
I20260620 23:09:40.997507 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:40.990739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:49617 (local address 127.12.2.65:37261)
0620 23:09:40.990964 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:40.990969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:40.991524 (+ 555us) server_negotiation.cc:407] Connection header received
0620 23:09:40.991668 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:40.991673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:40.991737 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:40.991809 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:40.992322 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.993391 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.995537 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:40.996631 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:40.996760 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:40.996856 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:40.997171 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 23:09:40.997294 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:09:40.997473 29715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e11d7cb61704bc58625432d0eb48941" candidate_uuid: "d05569a1c78a4c8ca163670b3ec9e184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19db49c648104c8da42af619064b7c61" is_pre_election: true
I20260620 23:09:40.998036 29715 raft_consensus.cc:2454] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d05569a1c78a4c8ca163670b3ec9e184 in term 0.
I20260620 23:09:40.998482 29581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e11d7cb61704bc58625432d0eb48941" candidate_uuid: "d05569a1c78a4c8ca163670b3ec9e184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019b469185284934b62c0529c969aba9" is_pre_election: true
I20260620 23:09:40.998729 29781 leader_election.cc:302] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19db49c648104c8da42af619064b7c61, d05569a1c78a4c8ca163670b3ec9e184; no voters:
I20260620 23:09:40.999022 29581 raft_consensus.cc:2454] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d05569a1c78a4c8ca163670b3ec9e184 in term 0.
I20260620 23:09:40.999146 29918 raft_consensus.cc:2781] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:09:40.999294 29918 raft_consensus.cc:482] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:09:40.999387 29918 raft_consensus.cc:3037] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:41.001551 29918 raft_consensus.cc:504] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19db49c648104c8da42af619064b7c61" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 35575 } } peers { permanent_uuid: "019b469185284934b62c0529c969aba9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37261 } } peers { permanent_uuid: "d05569a1c78a4c8ca163670b3ec9e184" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 46605 } }
I20260620 23:09:41.002105 29918 leader_election.cc:288] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [CANDIDATE]: Term 1 election: Requested vote from peers 19db49c648104c8da42af619064b7c61 (127.12.2.66:35575), 019b469185284934b62c0529c969aba9 (127.12.2.65:37261)
I20260620 23:09:41.002543 29581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e11d7cb61704bc58625432d0eb48941" candidate_uuid: "d05569a1c78a4c8ca163670b3ec9e184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019b469185284934b62c0529c969aba9"
I20260620 23:09:41.002542 29715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e11d7cb61704bc58625432d0eb48941" candidate_uuid: "d05569a1c78a4c8ca163670b3ec9e184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19db49c648104c8da42af619064b7c61"
I20260620 23:09:41.002720 29581 raft_consensus.cc:3037] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:41.002720 29715 raft_consensus.cc:3037] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:41.004851 29715 raft_consensus.cc:2454] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d05569a1c78a4c8ca163670b3ec9e184 in term 1.
I20260620 23:09:41.004999 29581 raft_consensus.cc:2454] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d05569a1c78a4c8ca163670b3ec9e184 in term 1.
I20260620 23:09:41.005208 29781 leader_election.cc:302] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19db49c648104c8da42af619064b7c61, d05569a1c78a4c8ca163670b3ec9e184; no voters:
I20260620 23:09:41.005498 29918 raft_consensus.cc:2781] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:41.005923 29918 raft_consensus.cc:686] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 1 LEADER]: Becoming Leader. State: Replica: d05569a1c78a4c8ca163670b3ec9e184, State: Running, Role: LEADER
I20260620 23:09:41.006457 29918 consensus_queue.cc:237] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19db49c648104c8da42af619064b7c61" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 35575 } } peers { permanent_uuid: "019b469185284934b62c0529c969aba9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37261 } } peers { permanent_uuid: "d05569a1c78a4c8ca163670b3ec9e184" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 46605 } }
I20260620 23:09:41.010802 29436 catalog_manager.cc:5495] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 reported cstate change: term changed from 0 to 1, leader changed from <none> to d05569a1c78a4c8ca163670b3ec9e184 (127.12.2.67). New cstate: current_term: 1 leader_uuid: "d05569a1c78a4c8ca163670b3ec9e184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19db49c648104c8da42af619064b7c61" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 35575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "019b469185284934b62c0529c969aba9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d05569a1c78a4c8ca163670b3ec9e184" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 46605 } health_report { overall_health: HEALTHY } } }
I20260620 23:09:41.076043 29910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:41.070764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.12.2.67:46605)
0620 23:09:41.071006 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:41.071010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:41.071028 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:09:41.071252 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:41.071256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:41.071356 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:41.071507 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:09:41.071820 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:41.072607 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:41.074025 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:41.074426 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:41.074540 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:41.074947 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:09:41.075521 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:09:41.075631 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:41.075713 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 23:09:41.075781 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:09:41.084219 29581 raft_consensus.cc:1261] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9 [term 1 FOLLOWER]: Refusing update from remote peer d05569a1c78a4c8ca163670b3ec9e184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:41.084219 29715 raft_consensus.cc:1261] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61 [term 1 FOLLOWER]: Refusing update from remote peer d05569a1c78a4c8ca163670b3ec9e184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:41.085314 29921 consensus_queue.cc:1035] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "019b469185284934b62c0529c969aba9" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:09:41.085690 29918 consensus_queue.cc:1035] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19db49c648104c8da42af619064b7c61" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 35575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:09:41.098158 29926 mvcc.cc:205] Tried to move back new op lower bound from 7299059634510848000 to 7299059634206507008. Current Snapshot: MvccSnapshot[applied={T|T < 7299059634510848000}]
I20260620 23:09:41.110419 29436 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:41.110688 29436 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:41.114013 29436 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dfac99fe5ae4b2cb42f08674149f077: Sending DeleteTablet for 3 replicas of tablet 2e11d7cb61704bc58625432d0eb48941
I20260620 23:09:41.115171 29828 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e11d7cb61704bc58625432d0eb48941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:41 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:55230
I20260620 23:09:41.115207 29561 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e11d7cb61704bc58625432d0eb48941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:41 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:35152
I20260620 23:09:41.116073 29695 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e11d7cb61704bc58625432d0eb48941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:41 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:60922
I20260620 23:09:41.116191 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 29498
I20260620 23:09:41.122741 29934 tablet_replica.cc:330] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: stopping tablet replica
I20260620 23:09:41.123733 29934 raft_consensus.cc:2229] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:41.124994 29934 raft_consensus.cc:2258] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:41.128226 29933 tablet_replica.cc:330] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: stopping tablet replica
I20260620 23:09:41.128666 29933 raft_consensus.cc:2229] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:09:41.129377 29933 raft_consensus.cc:2258] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:41.131366 29933 ts_tablet_manager.cc:1905] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:41.132198 29934 ts_tablet_manager.cc:1905] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:41.138574 29935 tablet_replica.cc:330] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: stopping tablet replica
I20260620 23:09:41.139011 29933 ts_tablet_manager.cc:1918] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:41.139053 29935 raft_consensus.cc:2229] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:41.139184 29933 log.cc:1196] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/2e11d7cb61704bc58625432d0eb48941
I20260620 23:09:41.139631 29935 raft_consensus.cc:2258] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:41.139657 29933 ts_tablet_manager.cc:1939] T 2e11d7cb61704bc58625432d0eb48941 P d05569a1c78a4c8ca163670b3ec9e184: Deleting consensus metadata
Jun 20 23:09:41 dist-test-slave-7g8g krb5kdc[29392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:09:41.141494 29423 catalog_manager.cc:4841] TS d05569a1c78a4c8ca163670b3ec9e184 (127.12.2.67:46605): tablet 2e11d7cb61704bc58625432d0eb48941 (table test-table [id=392f7ca62e1b430e880b77b7a9821d75]) successfully deleted
I20260620 23:09:41.141908 29935 ts_tablet_manager.cc:1905] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:41.144930 29934 ts_tablet_manager.cc:1918] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:41.145092 29934 log.cc:1196] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/2e11d7cb61704bc58625432d0eb48941
I20260620 23:09:41.145602 29934 ts_tablet_manager.cc:1939] T 2e11d7cb61704bc58625432d0eb48941 P 019b469185284934b62c0529c969aba9: Deleting consensus metadata
I20260620 23:09:41.147250 29422 catalog_manager.cc:4841] TS 019b469185284934b62c0529c969aba9 (127.12.2.65:37261): tablet 2e11d7cb61704bc58625432d0eb48941 (table test-table [id=392f7ca62e1b430e880b77b7a9821d75]) successfully deleted
I20260620 23:09:41.148809 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:41.122143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.12.2.65:37261)
0620 23:09:41.122349 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:41.122356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:41.129312 (+ 6956us) server_negotiation.cc:407] Connection header received
0620 23:09:41.129523 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:41.129529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:41.129613 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:41.129697 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:41.131770 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:41.132959 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:41.134920 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:41.135657 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:41.135759 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:41.141491 (+ 5732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:41.141509 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:41.141527 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:41.141584 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:41.145351 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:41.146217 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:41.146227 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:41.146230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:41.146395 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:41.146800 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:41.146805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:41.146808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:41.147641 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:41.147920 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:41.148241 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:09:41.148449 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:09:41.149480 29621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:41.153481 29935 ts_tablet_manager.cc:1918] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:41.153636 29935 log.cc:1196] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/2e11d7cb61704bc58625432d0eb48941
I20260620 23:09:41.154126 29935 ts_tablet_manager.cc:1939] T 2e11d7cb61704bc58625432d0eb48941 P 19db49c648104c8da42af619064b7c61: Deleting consensus metadata
I20260620 23:09:41.155893 29421 catalog_manager.cc:4841] TS 19db49c648104c8da42af619064b7c61 (127.12.2.66:35575): tablet 2e11d7cb61704bc58625432d0eb48941 (table test-table [id=392f7ca62e1b430e880b77b7a9821d75]) successfully deleted
I20260620 23:09:41.239298 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 29498
I20260620 23:09:41.254033 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 29632
Jun 20 23:09:41 dist-test-slave-7g8g krb5kdc[29392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:09:41.270529 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:41.255061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.12.2.66:35575)
0620 23:09:41.255207 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:41.255211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:41.255227 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:09:41.255281 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:41.255284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:41.255359 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:41.255454 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:41.257018 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:41.257770 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:41.259591 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:41.259962 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:41.260054 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:41.265017 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:41.265031 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:41.265048 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:41.265087 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:41.267977 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:41.268726 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:41.268733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:41.268735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:41.268849 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:41.269210 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:41.269213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:41.269215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:41.269567 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:41.269778 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:41.270100 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 23:09:41.270259 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:09:41.271291 29755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:41.393849 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 29632
I20260620 23:09:41.407536 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 29765
Jun 20 23:09:41 dist-test-slave-7g8g krb5kdc[29392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:09:41.423256 29910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:41.408393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.12.2.67:46605)
0620 23:09:41.408609 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:41.408614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:41.408638 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:09:41.408763 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:41.408768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:41.408842 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:41.408925 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:41.410219 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:41.411149 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:41.412372 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:41.412822 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:41.412940 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:41.417404 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:41.417415 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:41.417429 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:41.417469 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:41.420406 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:41.421206 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:41.421212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:41.421214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:41.421351 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:41.421722 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:41.421725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:41.421727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:41.422065 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:41.422267 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:41.422630 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 23:09:41.422786 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:09:41.423929 29888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:41.540720 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 29765
I20260620 23:09:41.555449 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 29405
I20260620 23:09:41.556608 29467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:41.663908 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 29405
2026-06-20T23:09:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4139 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T23:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:09:41Z Disabled control of system clock
I20260620 23:09:41.731534 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:44089
--webserver_interface=127.12.2.126
--webserver_port=0
--builtin_ntp_servers=127.12.2.84:46285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:44089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:09:42.043483 29944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:42.043763 29944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:42.043849 29944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:42.050954 29944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:09:42.051050 29944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:42.051093 29944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:42.051153 29944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:09:42.051194 29944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:09:42.060146 29944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:46285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:44089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:44089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:42.061367 29944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:42.063071 29944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:42.073896 29949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:42.073858 29952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:42.073892 29950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:42.074694 29944 server_base.cc:1034] running on GCE node
I20260620 23:09:42.075408 29944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:42.076699 29944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:42.077891 29944 hybrid_clock.cc:648] HybridClock initialized: now 1781996982077841 us; error 44 us; skew 500 ppm
I20260620 23:09:42.078255 29944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:42.080382 29944 webserver.cc:460] Webserver started at http://127.12.2.126:35195/ using document root <none> and password file <none>
I20260620 23:09:42.081014 29944 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:42.081099 29944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:42.081388 29944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:42.084167 29944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "4d6b2319a7d54248bbf3c57bb0d89dbe"
format_stamp: "Formatted at 2026-06-20 23:09:42 on dist-test-slave-7g8g"
server_key: "238ae264026453646d40564f0fc739cd"
server_key_iv: "c86313de2f20cd93f615993c10574964"
server_key_version: "encryptionkey@0"
I20260620 23:09:42.084913 29944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "4d6b2319a7d54248bbf3c57bb0d89dbe"
format_stamp: "Formatted at 2026-06-20 23:09:42 on dist-test-slave-7g8g"
server_key: "238ae264026453646d40564f0fc739cd"
server_key_iv: "c86313de2f20cd93f615993c10574964"
server_key_version: "encryptionkey@0"
I20260620 23:09:42.089807 29944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:09:42.093145 29958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:42.094445 29944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:09:42.094599 29944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "4d6b2319a7d54248bbf3c57bb0d89dbe"
format_stamp: "Formatted at 2026-06-20 23:09:42 on dist-test-slave-7g8g"
server_key: "238ae264026453646d40564f0fc739cd"
server_key_iv: "c86313de2f20cd93f615993c10574964"
server_key_version: "encryptionkey@0"
I20260620 23:09:42.094794 29944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:42.109099 29944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:42.109915 29944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:42.110191 29944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:42.127239 29944 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:44089
I20260620 23:09:42.127260 30009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:44089 every 8 connection(s)
I20260620 23:09:42.128533 29944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:09:42.131860 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 29944
I20260620 23:09:42.132064 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:09:42.132534 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:42.132604 12297 external_mini_cluster.cc:1428] Setting key 09a0c84e284e794e476a7c6525ed13e7
I20260620 23:09:42.144939 30010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe: Bootstrap starting.
I20260620 23:09:42.145605 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:42.134448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.12.2.126:44089)
0620 23:09:42.135418 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:42.135429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:42.135471 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:09:42.137282 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:42.137304 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:42.137793 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:42.138155 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:42.138745 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:42.140048 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:42.141382 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:42.142059 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:42.142222 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:42.143343 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:42.143350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:42.143363 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:42.143867 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:42.143945 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:42.143965 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:42.144066 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:42.144291 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 23:09:42.144577 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":340,"threads_started":1}
I20260620 23:09:42.148032 30010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:42.148993 30010 log.cc:824] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:42.151736 30010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe: No bootstrap required, opened a new log
I20260620 23:09:42.158959 30010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b2319a7d54248bbf3c57bb0d89dbe" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44089 } }
I20260620 23:09:42.159369 30010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:42.159479 30010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6b2319a7d54248bbf3c57bb0d89dbe, State: Initialized, Role: FOLLOWER
I20260620 23:09:42.160055 30010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b2319a7d54248bbf3c57bb0d89dbe" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44089 } }
I20260620 23:09:42.160274 30010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:09:42.160392 30010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:09:42.160552 30010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:42.162586 30010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b2319a7d54248bbf3c57bb0d89dbe" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44089 } }
I20260620 23:09:42.163116 30010 leader_election.cc:302] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d6b2319a7d54248bbf3c57bb0d89dbe; no voters:
I20260620 23:09:42.163640 30010 leader_election.cc:288] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:09:42.163770 30015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:42.164418 30015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [term 1 LEADER]: Becoming Leader. State: Replica: 4d6b2319a7d54248bbf3c57bb0d89dbe, State: Running, Role: LEADER
I20260620 23:09:42.164862 30015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b2319a7d54248bbf3c57bb0d89dbe" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44089 } }
I20260620 23:09:42.165608 30010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:09:42.167647 30017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d6b2319a7d54248bbf3c57bb0d89dbe. Latest consensus state: current_term: 1 leader_uuid: "4d6b2319a7d54248bbf3c57bb0d89dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b2319a7d54248bbf3c57bb0d89dbe" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44089 } } }
I20260620 23:09:42.167680 30016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d6b2319a7d54248bbf3c57bb0d89dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b2319a7d54248bbf3c57bb0d89dbe" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 44089 } } }
I20260620 23:09:42.168042 30017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:42.168123 30016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:42.170553 30022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:09:42.174907 30022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:09:42.183905 30022 catalog_manager.cc:1347] Generated new cluster ID: 99b31024b0b94e6fafaff8fddcbfa9fd
I20260620 23:09:42.184001 30022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:09:42.222635 30022 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:09:42.223938 30022 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:09:42.231848 30022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe: Generated new TSK 0
I20260620 23:09:42.232632 30022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:09:42.245133 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44089
--builtin_ntp_servers=127.12.2.84:46285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:09:42.555419 30034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:42.555680 30034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:42.555785 30034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:42.562925 30034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:42.563050 30034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:42.563196 30034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:09:42.572512 30034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:46285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:42.573817 30034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:42.575541 30034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:42.587935 30042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:42.587906 30040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:42.588029 30039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:42.588675 30034 server_base.cc:1034] running on GCE node
I20260620 23:09:42.589181 30034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:42.590070 30034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:42.591259 30034 hybrid_clock.cc:648] HybridClock initialized: now 1781996982591212 us; error 41 us; skew 500 ppm
I20260620 23:09:42.591539 30034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:42.596874 30034 webserver.cc:460] Webserver started at http://127.12.2.65:35575/ using document root <none> and password file <none>
I20260620 23:09:42.597508 30034 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:42.597612 30034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:42.597886 30034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:42.600767 30034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "23e68b16a4e446c19aceaf52c91a8c17"
format_stamp: "Formatted at 2026-06-20 23:09:42 on dist-test-slave-7g8g"
server_key: "0d7735bdfad151d69a667118dd721780"
server_key_iv: "8f87bed683c31e932a5c0dfeddb0afe2"
server_key_version: "encryptionkey@0"
I20260620 23:09:42.601504 30034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "23e68b16a4e446c19aceaf52c91a8c17"
format_stamp: "Formatted at 2026-06-20 23:09:42 on dist-test-slave-7g8g"
server_key: "0d7735bdfad151d69a667118dd721780"
server_key_iv: "8f87bed683c31e932a5c0dfeddb0afe2"
server_key_version: "encryptionkey@0"
I20260620 23:09:42.606094 30034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:09:42.609525 30048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:42.610746 30034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:09:42.610888 30034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "23e68b16a4e446c19aceaf52c91a8c17"
format_stamp: "Formatted at 2026-06-20 23:09:42 on dist-test-slave-7g8g"
server_key: "0d7735bdfad151d69a667118dd721780"
server_key_iv: "8f87bed683c31e932a5c0dfeddb0afe2"
server_key_version: "encryptionkey@0"
I20260620 23:09:42.611109 30034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:42.625705 30034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:42.626410 30034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:42.626646 30034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:42.627547 30034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:42.628854 30034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:42.628943 30034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:42.629045 30034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:42.629112 30034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:42.658002 30034 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:39079
I20260620 23:09:42.658046 30160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:39079 every 8 connection(s)
I20260620 23:09:42.659312 30034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:09:42.659857 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 30034
I20260620 23:09:42.660015 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:09:42.660564 12297 external_mini_cluster.cc:1428] Setting key 275d1f97d0fb7bfcb04c5b32f7583daa
I20260620 23:09:42.663727 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44089
--builtin_ntp_servers=127.12.2.84:46285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:09:42.673342 30163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:42.662009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:56257 (local address 127.12.2.126:44089)
0620 23:09:42.662430 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:42.662435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:42.663992 (+ 1557us) server_negotiation.cc:407] Connection header received
0620 23:09:42.666352 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:42.666358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:42.666460 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:42.666638 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:42.667870 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:42.668573 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:42.670609 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:42.671028 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:42.671163 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:42.672540 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:42.672546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:42.672550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:42.672572 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:42.672627 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:42.672644 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:42.672718 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:42.672922 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 23:09:42.673014 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":220,"threads_started":1}
I20260620 23:09:42.674285 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:42.662728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:44089 (local address 127.12.2.65:56257)
0620 23:09:42.663836 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:42.663885 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:42.666060 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:42.666852 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:42.666862 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:42.666996 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:42.667719 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:42.667742 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:42.668714 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:42.668719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:42.670468 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:42.670482 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:42.671250 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:42.671256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:42.671513 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:42.672205 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:09:42.672209 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:42.672271 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:09:42.672362 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:42.672732 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:42.672745 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:42.672753 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:09:42.672902 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 23:09:42.673264 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":330,"threads_started":1}
I20260620 23:09:42.675719 30161 heartbeater.cc:347] Connected to a master server at 127.12.2.126:44089
I20260620 23:09:42.676023 30161 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:42.676862 30161 heartbeater.cc:511] Master 127.12.2.126:44089 requested a full tablet report, sending...
I20260620 23:09:42.678833 29975 ts_manager.cc:194] Registered new tserver with Master: 23e68b16a4e446c19aceaf52c91a8c17 (127.12.2.65:39079)
I20260620 23:09:42.680258 29975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.65:56257
W20260620 23:09:42.975220 30165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:42.975492 30165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:42.975598 30165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:42.982842 30165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:42.982973 30165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:42.983127 30165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:09:42.992307 30165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:46285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:42.993569 30165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:42.995673 30165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:43.008586 30170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:43.008564 30171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:43.008641 30173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:43.009390 30165 server_base.cc:1034] running on GCE node
I20260620 23:09:43.009943 30165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:43.010843 30165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:43.012084 30165 hybrid_clock.cc:648] HybridClock initialized: now 1781996983012014 us; error 58 us; skew 500 ppm
I20260620 23:09:43.012396 30165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:43.014559 30165 webserver.cc:460] Webserver started at http://127.12.2.66:35281/ using document root <none> and password file <none>
I20260620 23:09:43.015275 30165 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:43.015373 30165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:43.015662 30165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:43.018457 30165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "e2b3c51808d249f2960404e35e23806f"
format_stamp: "Formatted at 2026-06-20 23:09:43 on dist-test-slave-7g8g"
server_key: "3789929f131f6daff522b812f24aa3e4"
server_key_iv: "088c9a527a247f1a0cdfd859192094ce"
server_key_version: "encryptionkey@0"
I20260620 23:09:43.019246 30165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "e2b3c51808d249f2960404e35e23806f"
format_stamp: "Formatted at 2026-06-20 23:09:43 on dist-test-slave-7g8g"
server_key: "3789929f131f6daff522b812f24aa3e4"
server_key_iv: "088c9a527a247f1a0cdfd859192094ce"
server_key_version: "encryptionkey@0"
I20260620 23:09:43.024148 30165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 23:09:43.027546 30179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:43.028834 30165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:09:43.028975 30165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "e2b3c51808d249f2960404e35e23806f"
format_stamp: "Formatted at 2026-06-20 23:09:43 on dist-test-slave-7g8g"
server_key: "3789929f131f6daff522b812f24aa3e4"
server_key_iv: "088c9a527a247f1a0cdfd859192094ce"
server_key_version: "encryptionkey@0"
I20260620 23:09:43.029173 30165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:43.053534 30165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:43.054358 30165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:43.054610 30165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:43.055621 30165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:43.057090 30165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:43.057180 30165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:43.057286 30165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:43.057354 30165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:43.086570 30165 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:40977
I20260620 23:09:43.086625 30291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:40977 every 8 connection(s)
I20260620 23:09:43.087855 30165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:09:43.090963 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 30165
I20260620 23:09:43.091182 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:09:43.091754 12297 external_mini_cluster.cc:1428] Setting key 1da3b8b539354785df089238d86089ce
I20260620 23:09:43.094573 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44089
--builtin_ntp_servers=127.12.2.84:46285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:09:43.100567 30163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.090362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:52885 (local address 127.12.2.126:44089)
0620 23:09:43.090563 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:43.090568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:43.092275 (+ 1707us) server_negotiation.cc:407] Connection header received
0620 23:09:43.094399 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:43.094404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:43.094498 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:43.094580 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:43.095664 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.096377 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.097901 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.098394 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.098514 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.099781 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:43.099787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:43.099790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:43.099807 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:43.099877 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:43.099892 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:43.099966 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:43.100203 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:09:43.100289 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:09:43.101715 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.091054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:44089 (local address 127.12.2.66:52885)
0620 23:09:43.092108 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:43.092147 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:43.094148 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:43.094809 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:43.094817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:43.094911 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:43.095510 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.095527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.096528 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.096532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.097768 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.097780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.098599 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.098607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.098823 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.099478 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:09:43.099480 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:43.099519 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:09:43.099599 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:43.099998 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:43.100009 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:43.100017 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:09:43.100187 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 23:09:43.100551 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":296,"threads_started":1}
I20260620 23:09:43.103024 30292 heartbeater.cc:347] Connected to a master server at 127.12.2.126:44089
I20260620 23:09:43.103325 30292 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:43.104153 30292 heartbeater.cc:511] Master 127.12.2.126:44089 requested a full tablet report, sending...
I20260620 23:09:43.105554 29975 ts_manager.cc:194] Registered new tserver with Master: e2b3c51808d249f2960404e35e23806f (127.12.2.66:40977)
I20260620 23:09:43.106544 29975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.66:52885
W20260620 23:09:43.419503 30295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:43.419764 30295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:43.419853 30295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:43.426849 30295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:43.426970 30295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:43.427099 30295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:09:43.435868 30295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:46285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:44089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:43.437074 30295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:43.438738 30295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:43.450688 30303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:43.450862 30300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:43.450764 30301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:43.451673 30295 server_base.cc:1034] running on GCE node
I20260620 23:09:43.452224 30295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:43.453078 30295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:43.454304 30295 hybrid_clock.cc:648] HybridClock initialized: now 1781996983454248 us; error 43 us; skew 500 ppm
I20260620 23:09:43.454631 30295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:43.456719 30295 webserver.cc:460] Webserver started at http://127.12.2.67:39461/ using document root <none> and password file <none>
I20260620 23:09:43.457376 30295 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:43.457489 30295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:43.457764 30295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:43.460491 30295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8"
format_stamp: "Formatted at 2026-06-20 23:09:43 on dist-test-slave-7g8g"
server_key: "592b41990443d8cb1033aafdd627e309"
server_key_iv: "9deb472f3bc89367f3f471a3016d908b"
server_key_version: "encryptionkey@0"
I20260620 23:09:43.461210 30295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8"
format_stamp: "Formatted at 2026-06-20 23:09:43 on dist-test-slave-7g8g"
server_key: "592b41990443d8cb1033aafdd627e309"
server_key_iv: "9deb472f3bc89367f3f471a3016d908b"
server_key_version: "encryptionkey@0"
I20260620 23:09:43.465991 30295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 23:09:43.469285 30309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:43.470491 30295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:09:43.470633 30295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8"
format_stamp: "Formatted at 2026-06-20 23:09:43 on dist-test-slave-7g8g"
server_key: "592b41990443d8cb1033aafdd627e309"
server_key_iv: "9deb472f3bc89367f3f471a3016d908b"
server_key_version: "encryptionkey@0"
I20260620 23:09:43.470809 30295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:43.487457 30295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:43.488210 30295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:43.488456 30295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:43.489315 30295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:43.490650 30295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:43.490751 30295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:43.490845 30295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:43.490911 30295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:43.519533 30295 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:45995
I20260620 23:09:43.519588 30421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:45995 every 8 connection(s)
I20260620 23:09:43.520762 30295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:09:43.528458 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 30295
I20260620 23:09:43.528646 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:09:43.529055 12297 external_mini_cluster.cc:1428] Setting key 73016bb32e69f2e13a1980d7fc0dc923
I20260620 23:09:43.533136 30163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.523079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:51659 (local address 127.12.2.126:44089)
0620 23:09:43.523251 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:43.523255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:43.524588 (+ 1333us) server_negotiation.cc:407] Connection header received
0620 23:09:43.526285 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:43.526290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:43.526368 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:43.526453 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:43.527568 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.528415 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.530377 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.530920 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.531052 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.532492 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:43.532496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:43.532499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:43.532520 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:43.532551 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:43.532562 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:43.532626 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:43.532815 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 23:09:43.532890 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:09:43.534199 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.523585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:44089 (local address 127.12.2.67:51659)
0620 23:09:43.524431 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:43.524471 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:43.526024 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:43.526691 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:43.526699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:43.526781 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:43.527410 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.527428 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.528608 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.528614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.530232 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.530247 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.531153 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.531160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.531454 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.532177 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:09:43.532181 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:43.532230 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:09:43.532326 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:43.532648 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:43.532656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:43.532662 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:09:43.532804 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 23:09:43.533127 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":256,"threads_started":1}
I20260620 23:09:43.535442 30422 heartbeater.cc:347] Connected to a master server at 127.12.2.126:44089
I20260620 23:09:43.535714 30422 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:43.536428 30422 heartbeater.cc:511] Master 127.12.2.126:44089 requested a full tablet report, sending...
I20260620 23:09:43.537777 29975 ts_manager.cc:194] Registered new tserver with Master: d2ee77ab3d2d42bb9b1ae22829305ae8 (127.12.2.67:45995)
I20260620 23:09:43.538589 29975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.67:51659
I20260620 23:09:43.543828 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:09:43.552788 30163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.547800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.12.2.126:44089)
0620 23:09:43.547969 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:43.547973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:43.548308 (+ 335us) server_negotiation.cc:407] Connection header received
0620 23:09:43.548441 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:43.548444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:43.548516 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:43.548597 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:43.548982 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.549657 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.550907 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.551331 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.551422 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.552282 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:43.552287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:43.552290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:43.552307 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:43.552337 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:43.552348 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:43.552413 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:43.552494 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 23:09:43.552564 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:09:43.556164 29975 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:09:43.562778 29975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:09:43.591842 30436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.574435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:40977 (local address 127.0.0.1:44874)
0620 23:09:43.583382 (+ 8947us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:43.583405 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:43.583558 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:43.584267 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:43.584275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:43.584304 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:43.584674 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.584687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.586644 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.586650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.588780 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.588795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.590915 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.590922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.591135 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.591187 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:09:43.591301 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:09:43.591480 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8798,"mutex_wait_us":22,"thread_start_us":8669,"threads_started":1}
I20260620 23:09:43.592625 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.575024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.12.2.66:40977)
0620 23:09:43.581476 (+ 6452us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:43.581483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:43.583701 (+ 2218us) server_negotiation.cc:407] Connection header received
0620 23:09:43.583817 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:43.583825 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:43.584062 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:43.584240 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:43.584860 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.586493 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.588953 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.590397 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.590522 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.590803 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:43.592093 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0620 23:09:43.592263 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6284,"thread_start_us":6156,"threads_started":1}
I20260620 23:09:43.593129 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.574236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.12.2.67:45995)
0620 23:09:43.586036 (+ 11800us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:43.586041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:43.586066 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:09:43.586146 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:43.586153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:43.586363 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:43.586552 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:43.587269 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.588724 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.590506 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.591839 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.592235 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.592498 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:43.592621 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 23:09:43.592785 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11646,"thread_start_us":11509,"threads_started":1}
I20260620 23:09:43.593470 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.573972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:45995 (local address 127.0.0.1:33860)
0620 23:09:43.574768 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:43.574818 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:43.575020 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:43.586564 (+ 11544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:43.586569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:43.586595 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:43.587092 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.587104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.588860 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.588865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.590327 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.590339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.592253 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.592259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.592430 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.592463 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:09:43.592563 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:09:43.592874 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":42,"thread_start_us":374,"threads_started":1}
I20260620 23:09:43.593930 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.575252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39079 (local address 127.0.0.1:35900)
0620 23:09:43.583667 (+ 8415us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:43.583682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:43.583772 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:43.586171 (+ 2399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:43.586176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:43.586201 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:43.586554 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.586568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.588328 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.588334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.590542 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.590556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.592579 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.592586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.593447 (+ 861us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.593487 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:09:43.593572 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:09:43.593686 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8309,"mutex_wait_us":67,"thread_start_us":8196,"threads_started":1}
I20260620 23:09:43.594561 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.575623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.12.2.65:39079)
0620 23:09:43.585599 (+ 9976us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:43.585607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:43.585636 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:09:43.585724 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:43.585731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:43.585985 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:43.586167 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:43.586772 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.588179 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.590716 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.592110 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.592227 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.592449 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:43.594182 (+ 1733us) server_negotiation.cc:299] Negotiation successful
0620 23:09:43.594313 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9814,"thread_start_us":9723,"threads_started":1}
I20260620 23:09:43.596764 30227 tablet_service.cc:1467] Processing CreateTablet for tablet ded324a486a344a3b3c3f36c37dd0669 (DEFAULT_TABLE table=test-table [id=72aef0655487415ea382a791409bb817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:43.597054 30357 tablet_service.cc:1467] Processing CreateTablet for tablet ded324a486a344a3b3c3f36c37dd0669 (DEFAULT_TABLE table=test-table [id=72aef0655487415ea382a791409bb817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:43.598256 30227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded324a486a344a3b3c3f36c37dd0669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:43.598256 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded324a486a344a3b3c3f36c37dd0669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:43.597860 30096 tablet_service.cc:1467] Processing CreateTablet for tablet ded324a486a344a3b3c3f36c37dd0669 (DEFAULT_TABLE table=test-table [id=72aef0655487415ea382a791409bb817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:43.599118 30096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded324a486a344a3b3c3f36c37dd0669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:43.609531 30440 tablet_bootstrap.cc:492] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8: Bootstrap starting.
I20260620 23:09:43.611790 30441 tablet_bootstrap.cc:492] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f: Bootstrap starting.
I20260620 23:09:43.613162 30440 tablet_bootstrap.cc:654] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:43.614562 30440 log.cc:824] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:43.615017 30441 tablet_bootstrap.cc:654] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:43.616219 30441 log.cc:824] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:43.617779 30442 tablet_bootstrap.cc:492] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17: Bootstrap starting.
I20260620 23:09:43.619321 30441 tablet_bootstrap.cc:492] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f: No bootstrap required, opened a new log
I20260620 23:09:43.619580 30441 ts_tablet_manager.cc:1397] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 23:09:43.620443 30440 tablet_bootstrap.cc:492] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8: No bootstrap required, opened a new log
I20260620 23:09:43.620679 30440 ts_tablet_manager.cc:1397] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 23:09:43.621060 30442 tablet_bootstrap.cc:654] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:43.622145 30442 log.cc:824] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:43.627684 30440 raft_consensus.cc:348] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45995 } } peers { permanent_uuid: "23e68b16a4e446c19aceaf52c91a8c17" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39079 } } peers { permanent_uuid: "e2b3c51808d249f2960404e35e23806f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40977 } }
I20260620 23:09:43.628099 30440 raft_consensus.cc:374] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:43.628187 30440 raft_consensus.cc:729] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ee77ab3d2d42bb9b1ae22829305ae8, State: Initialized, Role: FOLLOWER
I20260620 23:09:43.628697 30440 consensus_queue.cc:260] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45995 } } peers { permanent_uuid: "23e68b16a4e446c19aceaf52c91a8c17" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39079 } } peers { permanent_uuid: "e2b3c51808d249f2960404e35e23806f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40977 } }
I20260620 23:09:43.629679 30442 tablet_bootstrap.cc:492] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17: No bootstrap required, opened a new log
I20260620 23:09:43.629666 30422 heartbeater.cc:503] Master 127.12.2.126:44089 was elected leader, sending a full tablet report...
I20260620 23:09:43.629930 30442 ts_tablet_manager.cc:1397] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260620 23:09:43.630148 30440 ts_tablet_manager.cc:1428] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20260620 23:09:43.629344 30441 raft_consensus.cc:348] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45995 } } peers { permanent_uuid: "23e68b16a4e446c19aceaf52c91a8c17" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39079 } } peers { permanent_uuid: "e2b3c51808d249f2960404e35e23806f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40977 } }
I20260620 23:09:43.631245 30441 raft_consensus.cc:374] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:43.631347 30441 raft_consensus.cc:729] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b3c51808d249f2960404e35e23806f, State: Initialized, Role: FOLLOWER
I20260620 23:09:43.631902 30441 consensus_queue.cc:260] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45995 } } peers { permanent_uuid: "23e68b16a4e446c19aceaf52c91a8c17" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39079 } } peers { permanent_uuid: "e2b3c51808d249f2960404e35e23806f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40977 } }
I20260620 23:09:43.639714 30292 heartbeater.cc:503] Master 127.12.2.126:44089 was elected leader, sending a full tablet report...
I20260620 23:09:43.640398 30442 raft_consensus.cc:348] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45995 } } peers { permanent_uuid: "23e68b16a4e446c19aceaf52c91a8c17" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39079 } } peers { permanent_uuid: "e2b3c51808d249f2960404e35e23806f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40977 } }
I20260620 23:09:43.640740 30441 ts_tablet_manager.cc:1428] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f: Time spent starting tablet: real 0.021s user 0.014s sys 0.004s
I20260620 23:09:43.640858 30442 raft_consensus.cc:374] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:43.640956 30442 raft_consensus.cc:729] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e68b16a4e446c19aceaf52c91a8c17, State: Initialized, Role: FOLLOWER
I20260620 23:09:43.641481 30442 consensus_queue.cc:260] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45995 } } peers { permanent_uuid: "23e68b16a4e446c19aceaf52c91a8c17" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39079 } } peers { permanent_uuid: "e2b3c51808d249f2960404e35e23806f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40977 } }
I20260620 23:09:43.642478 30161 heartbeater.cc:503] Master 127.12.2.126:44089 was elected leader, sending a full tablet report...
I20260620 23:09:43.643011 30442 ts_tablet_manager.cc:1428] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260620 23:09:43.659096 30447 raft_consensus.cc:482] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:09:43.659336 30447 raft_consensus.cc:504] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45995 } } peers { permanent_uuid: "23e68b16a4e446c19aceaf52c91a8c17" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39079 } } peers { permanent_uuid: "e2b3c51808d249f2960404e35e23806f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40977 } }
I20260620 23:09:43.660812 30447 leader_election.cc:288] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2ee77ab3d2d42bb9b1ae22829305ae8 (127.12.2.67:45995), 23e68b16a4e446c19aceaf52c91a8c17 (127.12.2.65:39079)
W20260620 23:09:43.661840 30162 tablet.cc:2369] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:09:43.668489 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.661307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:50165 (local address 127.12.2.67:45995)
0620 23:09:43.661564 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:43.661570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:43.661955 (+ 385us) server_negotiation.cc:407] Connection header received
0620 23:09:43.662332 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:43.662337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:43.662413 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:43.662474 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:43.663322 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.664263 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.666890 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.667708 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.667820 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.667915 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:43.668152 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:09:43.668270 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 23:09:43.668614 30449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.661211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:45995 (local address 127.12.2.66:50165)
0620 23:09:43.661857 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:43.661876 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:43.662160 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:43.662668 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:43.662672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:43.662697 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:43.663153 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.663168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.664392 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.664398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.666678 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.666692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.667882 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.667887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.668018 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.668042 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:09:43.668118 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:09:43.668236 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":1,"thread_start_us":357,"threads_started":1}
I20260620 23:09:43.669464 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.662272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:60117 (local address 127.12.2.65:39079)
0620 23:09:43.662420 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:43.662425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:43.663703 (+ 1278us) server_negotiation.cc:407] Connection header received
0620 23:09:43.663847 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:43.663852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:43.663948 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:43.664088 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:43.664611 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.665313 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.667043 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.668242 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.668358 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.668462 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:43.669100 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 23:09:43.669225 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:09:43.669478 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.661953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39079 (local address 127.12.2.66:60117)
0620 23:09:43.663603 (+ 1650us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:43.663620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:43.663734 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:43.664103 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:43.664108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:43.664131 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:43.664462 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.664472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.665435 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.665439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.666848 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:43.666863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.668610 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.668616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:43.668924 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.668977 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 23:09:43.669058 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:09:43.669168 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1531,"mutex_wait_us":34,"thread_start_us":1377,"threads_started":1}
I20260620 23:09:43.669524 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded324a486a344a3b3c3f36c37dd0669" candidate_uuid: "e2b3c51808d249f2960404e35e23806f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" is_pre_election: true
I20260620 23:09:43.670090 30377 raft_consensus.cc:2454] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b3c51808d249f2960404e35e23806f in term 0.
I20260620 23:09:43.670423 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded324a486a344a3b3c3f36c37dd0669" candidate_uuid: "e2b3c51808d249f2960404e35e23806f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23e68b16a4e446c19aceaf52c91a8c17" is_pre_election: true
I20260620 23:09:43.670816 30180 leader_election.cc:302] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2ee77ab3d2d42bb9b1ae22829305ae8, e2b3c51808d249f2960404e35e23806f; no voters:
I20260620 23:09:43.670892 30116 raft_consensus.cc:2454] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b3c51808d249f2960404e35e23806f in term 0.
I20260620 23:09:43.671216 30447 raft_consensus.cc:2781] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:09:43.671378 30447 raft_consensus.cc:482] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:09:43.671491 30447 raft_consensus.cc:3037] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:43.673651 30447 raft_consensus.cc:504] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45995 } } peers { permanent_uuid: "23e68b16a4e446c19aceaf52c91a8c17" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39079 } } peers { permanent_uuid: "e2b3c51808d249f2960404e35e23806f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40977 } }
I20260620 23:09:43.674197 30447 leader_election.cc:288] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [CANDIDATE]: Term 1 election: Requested vote from peers d2ee77ab3d2d42bb9b1ae22829305ae8 (127.12.2.67:45995), 23e68b16a4e446c19aceaf52c91a8c17 (127.12.2.65:39079)
I20260620 23:09:43.674635 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded324a486a344a3b3c3f36c37dd0669" candidate_uuid: "e2b3c51808d249f2960404e35e23806f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8"
I20260620 23:09:43.674763 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded324a486a344a3b3c3f36c37dd0669" candidate_uuid: "e2b3c51808d249f2960404e35e23806f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23e68b16a4e446c19aceaf52c91a8c17"
I20260620 23:09:43.674830 30377 raft_consensus.cc:3037] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:43.674973 30116 raft_consensus.cc:3037] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:43.677889 30377 raft_consensus.cc:2454] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b3c51808d249f2960404e35e23806f in term 1.
I20260620 23:09:43.677889 30116 raft_consensus.cc:2454] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b3c51808d249f2960404e35e23806f in term 1.
I20260620 23:09:43.678226 30180 leader_election.cc:302] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2ee77ab3d2d42bb9b1ae22829305ae8, e2b3c51808d249f2960404e35e23806f; no voters:
I20260620 23:09:43.678479 30447 raft_consensus.cc:2781] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:43.678980 30447 raft_consensus.cc:686] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 1 LEADER]: Becoming Leader. State: Replica: e2b3c51808d249f2960404e35e23806f, State: Running, Role: LEADER
I20260620 23:09:43.679447 30447 consensus_queue.cc:237] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45995 } } peers { permanent_uuid: "23e68b16a4e446c19aceaf52c91a8c17" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39079 } } peers { permanent_uuid: "e2b3c51808d249f2960404e35e23806f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40977 } }
I20260620 23:09:43.684031 29975 catalog_manager.cc:5495] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f reported cstate change: term changed from 0 to 1, leader changed from <none> to e2b3c51808d249f2960404e35e23806f (127.12.2.66). New cstate: current_term: 1 leader_uuid: "e2b3c51808d249f2960404e35e23806f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23e68b16a4e446c19aceaf52c91a8c17" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2b3c51808d249f2960404e35e23806f" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40977 } health_report { overall_health: HEALTHY } } }
I20260620 23:09:43.718133 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.712784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.12.2.66:40977)
0620 23:09:43.713017 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:43.713023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:43.713063 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:09:43.713291 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:43.713295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:43.713378 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:43.713508 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:09:43.713855 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.714585 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.716085 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.716495 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.716603 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.717097 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:09:43.717587 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:09:43.717721 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:43.717820 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 23:09:43.717900 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:09:43.726177 30377 raft_consensus.cc:1261] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8 [term 1 FOLLOWER]: Refusing update from remote peer e2b3c51808d249f2960404e35e23806f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:43.726176 30116 raft_consensus.cc:1261] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17 [term 1 FOLLOWER]: Refusing update from remote peer e2b3c51808d249f2960404e35e23806f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:43.727205 30451 consensus_queue.cc:1035] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2ee77ab3d2d42bb9b1ae22829305ae8" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:09:43.727535 30447 consensus_queue.cc:1035] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [LEADER]: Connected to new peer: Peer: permanent_uuid: "23e68b16a4e446c19aceaf52c91a8c17" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:09:43.740423 30456 mvcc.cc:205] Tried to move back new op lower bound from 7299059645332271104 to 7299059645155123200. Current Snapshot: MvccSnapshot[applied={T|T < 7299059645332271104}]
I20260620 23:09:43.754159 29974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:43.754400 29974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:43.758028 29974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d6b2319a7d54248bbf3c57bb0d89dbe: Sending DeleteTablet for 3 replicas of tablet ded324a486a344a3b3c3f36c37dd0669
I20260620 23:09:43.759078 30357 tablet_service.cc:1514] Processing DeleteTablet for tablet ded324a486a344a3b3c3f36c37dd0669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:43 UTC) from {username='slave'} at 127.0.0.1:33860
I20260620 23:09:43.759163 30096 tablet_service.cc:1514] Processing DeleteTablet for tablet ded324a486a344a3b3c3f36c37dd0669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:43 UTC) from {username='slave'} at 127.0.0.1:35900
I20260620 23:09:43.759536 30227 tablet_service.cc:1514] Processing DeleteTablet for tablet ded324a486a344a3b3c3f36c37dd0669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:43 UTC) from {username='slave'} at 127.0.0.1:44874
I20260620 23:09:43.760365 30466 tablet_replica.cc:330] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f: stopping tablet replica
I20260620 23:09:43.760725 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 30034
I20260620 23:09:43.760838 30466 raft_consensus.cc:2229] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:09:43.761564 30466 raft_consensus.cc:2258] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:43.764302 30466 ts_tablet_manager.cc:1905] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:43.769848 30464 tablet_replica.cc:330] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8: stopping tablet replica
I20260620 23:09:43.770835 30464 raft_consensus.cc:2229] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:43.771436 30464 raft_consensus.cc:2258] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:43.771646 30466 ts_tablet_manager.cc:1918] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:43.771879 30466 log.cc:1196] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/ded324a486a344a3b3c3f36c37dd0669
I20260620 23:09:43.772604 30466 ts_tablet_manager.cc:1939] T ded324a486a344a3b3c3f36c37dd0669 P e2b3c51808d249f2960404e35e23806f: Deleting consensus metadata
I20260620 23:09:43.772769 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.762688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.12.2.65:39079)
0620 23:09:43.762865 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:43.762871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:43.762890 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:09:43.762979 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:43.762985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:43.763093 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:43.763193 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:43.765370 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.766615 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.768693 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.769386 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.769477 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.770870 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:43.770881 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:43.770897 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:43.771731 (+ 834us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:43.771786 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:43.771810 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:43.771885 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:43.772305 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 23:09:43.772417 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:09:43.773690 30464 ts_tablet_manager.cc:1905] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:43.773902 30156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:43.773952 30465 tablet_replica.cc:330] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17: stopping tablet replica
I20260620 23:09:43.775033 29960 catalog_manager.cc:4841] TS e2b3c51808d249f2960404e35e23806f (127.12.2.66:40977): tablet ded324a486a344a3b3c3f36c37dd0669 (table test-table [id=72aef0655487415ea382a791409bb817]) successfully deleted
I20260620 23:09:43.780577 30464 ts_tablet_manager.cc:1918] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:43.780735 30464 log.cc:1196] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/ded324a486a344a3b3c3f36c37dd0669
I20260620 23:09:43.781476 30464 ts_tablet_manager.cc:1939] T ded324a486a344a3b3c3f36c37dd0669 P d2ee77ab3d2d42bb9b1ae22829305ae8: Deleting consensus metadata
I20260620 23:09:43.783627 29959 catalog_manager.cc:4841] TS d2ee77ab3d2d42bb9b1ae22829305ae8 (127.12.2.67:45995): tablet ded324a486a344a3b3c3f36c37dd0669 (table test-table [id=72aef0655487415ea382a791409bb817]) successfully deleted
I20260620 23:09:43.774345 30465 raft_consensus.cc:2229] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:43.865960 30465 raft_consensus.cc:2258] T ded324a486a344a3b3c3f36c37dd0669 P 23e68b16a4e446c19aceaf52c91a8c17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:43.866125 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 30034
I20260620 23:09:43.880705 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 30165
W20260620 23:09:43.880658 29959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.2.65:39079 (error 108)
I20260620 23:09:43.882678 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.882259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39079 (local address 127.0.0.1:35920)
0620 23:09:43.882424 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:43.882544 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:09:43.882963 29959 catalog_manager.cc:4584] TS 23e68b16a4e446c19aceaf52c91a8c17 (127.12.2.65:39079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ded324a486a344a3b3c3f36c37dd0669: Network error: Client connection negotiation failed: client connection to 127.12.2.65:39079: connect: Connection refused (error 111)
I20260620 23:09:43.888976 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.881911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.12.2.66:40977)
0620 23:09:43.882060 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:43.882064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:43.882080 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:09:43.882133 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:43.882136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:43.882232 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:43.882334 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:43.882965 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.884112 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.885416 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:43.886062 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:43.886206 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:43.887350 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:43.887363 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:43.887383 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:43.888188 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:43.888248 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:43.888277 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:43.888378 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:43.888507 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 23:09:43.888608 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:09:43.889704 30287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:43.936694 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.936042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39079 (local address 127.0.0.1:35934)
0620 23:09:43.936326 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:43.936462 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 23:09:43.937815 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.937389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39079 (local address 127.0.0.1:35940)
0620 23:09:43.937552 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:43.937647 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 23:09:43.997025 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 30165
I20260620 23:09:43.999840 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:43.997951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39079 (local address 127.0.0.1:35948)
0620 23:09:43.999524 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:43.999640 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1412}
I20260620 23:09:44.000969 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:44.000582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39079 (local address 127.0.0.1:35954)
0620 23:09:44.000768 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:44.000843 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 23:09:44.013314 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 30295
I20260620 23:09:44.019480 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:44.014097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.12.2.67:45995)
0620 23:09:44.014249 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:44.014254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:44.014271 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:09:44.014369 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:44.014373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:44.014454 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:44.014526 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:44.014897 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:44.015656 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:44.016879 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:44.017251 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:44.017338 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:44.018258 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:44.018265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:44.018279 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:44.018871 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:44.018914 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:44.018965 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:44.019054 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:44.019153 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 23:09:44.019238 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:09:44.020057 30417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:44.081457 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:44.080906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39079 (local address 127.0.0.1:35968)
0620 23:09:44.081115 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:44.081224 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 23:09:44.082547 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:44.082110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39079 (local address 127.0.0.1:35976)
0620 23:09:44.082317 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:44.082411 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 23:09:44.107159 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 30295
I20260620 23:09:44.119971 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 29944
I20260620 23:09:44.120934 30005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:44.227161 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 29944
2026-06-20T23:09:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2564 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:09:44 dist-test-slave-7g8g krb5kdc[30475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:09:44 dist-test-slave-7g8g krb5kdc[30475](info): set up 2 sockets
Jun 20 23:09:44 dist-test-slave-7g8g krb5kdc[30475](info): commencing operation
krb5kdc: starting...
W20260620 23:09:45.242818 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:09:45 dist-test-slave-7g8g krb5kdc[30475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:09:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:09:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:09:45.464016 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:40103
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:40103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:45.775262 30488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:45.775537 30488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:45.775645 30488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:45.782740 30488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:09:45.782846 30488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:45.782891 30488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:45.782974 30488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:09:45.783031 30488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:09:45.792722 30488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:40103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:40103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:45.793991 30488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:45.795312 30488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:45.806514 30493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:45.806461 30496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:45.806545 30494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:45.807407 30488 server_base.cc:1034] running on GCE node
I20260620 23:09:45.808135 30488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:45.809489 30488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:45.810698 30488 hybrid_clock.cc:648] HybridClock initialized: now 1781996985810646 us; error 39 us; skew 500 ppm
I20260620 23:09:45.811086 30488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:45 dist-test-slave-7g8g krb5kdc[30475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996985, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:45.814776 30488 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:09:45.819716 30488 webserver.cc:460] Webserver started at http://127.12.2.126:37549/ using document root <none> and password file <none>
I20260620 23:09:45.820770 30488 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:45.820911 30488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:45.821288 30488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:45.825690 30488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "18426c99f3d64fe2b96324d47fcf5381"
format_stamp: "Formatted at 2026-06-20 23:09:45 on dist-test-slave-7g8g"
server_key: "aab0812b12446d3a5b65332d31fce6a6"
server_key_iv: "22e579646e971fe1b2c483199d60da5d"
server_key_version: "encryptionkey@0"
I20260620 23:09:45.826704 30488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "18426c99f3d64fe2b96324d47fcf5381"
format_stamp: "Formatted at 2026-06-20 23:09:45 on dist-test-slave-7g8g"
server_key: "aab0812b12446d3a5b65332d31fce6a6"
server_key_iv: "22e579646e971fe1b2c483199d60da5d"
server_key_version: "encryptionkey@0"
I20260620 23:09:45.833096 30488 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260620 23:09:45.837061 30503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:45.838472 30488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:09:45.838641 30488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "18426c99f3d64fe2b96324d47fcf5381"
format_stamp: "Formatted at 2026-06-20 23:09:45 on dist-test-slave-7g8g"
server_key: "aab0812b12446d3a5b65332d31fce6a6"
server_key_iv: "22e579646e971fe1b2c483199d60da5d"
server_key_version: "encryptionkey@0"
I20260620 23:09:45.838835 30488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:45.864516 30488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:45.868698 30488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:45.868923 30488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:45.886016 30488 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:40103
I20260620 23:09:45.886081 30554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:40103 every 8 connection(s)
I20260620 23:09:45.887527 30488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:09:45.892382 30555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:45.895438 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 30488
I20260620 23:09:45.895614 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:09:45.896145 12297 external_mini_cluster.cc:1428] Setting key 809aab01386e4710714f19071bd6cc8c
I20260620 23:09:45.904731 30555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381: Bootstrap starting.
I20260620 23:09:45.907842 30555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:45.908775 30555 log.cc:824] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:09:45 dist-test-slave-7g8g krb5kdc[30475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:45.911685 30555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381: No bootstrap required, opened a new log
I20260620 23:09:45.917389 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:45.898064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.12.2.126:40103)
0620 23:09:45.898800 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:45.898815 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:45.898864 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:09:45.900570 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:45.900593 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:45.900980 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:45.901348 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:45.902758 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:45.904153 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:45.905482 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:45.906162 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:45.906328 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:45.910354 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:45.910361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:45.910375 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:45.910413 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:45.913437 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:45.914116 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:45.914123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:45.914130 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:45.914291 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:45.914663 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:45.914666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:45.914668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:45.915236 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:45.915539 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:45.915969 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 23:09:45.916372 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":284,"threads_started":1}
I20260620 23:09:45.919824 30555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18426c99f3d64fe2b96324d47fcf5381" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40103 } }
I20260620 23:09:45.920187 30555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:45.920292 30555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18426c99f3d64fe2b96324d47fcf5381, State: Initialized, Role: FOLLOWER
I20260620 23:09:45.920818 30555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18426c99f3d64fe2b96324d47fcf5381" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40103 } }
I20260620 23:09:45.921066 30555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:09:45.921185 30555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:09:45.921334 30555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:45.923466 30555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18426c99f3d64fe2b96324d47fcf5381" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40103 } }
I20260620 23:09:45.923923 30555 leader_election.cc:302] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18426c99f3d64fe2b96324d47fcf5381; no voters:
I20260620 23:09:45.924440 30555 leader_election.cc:288] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:09:45.924546 30560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:45.925173 30560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [term 1 LEADER]: Becoming Leader. State: Replica: 18426c99f3d64fe2b96324d47fcf5381, State: Running, Role: LEADER
I20260620 23:09:45.925670 30560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18426c99f3d64fe2b96324d47fcf5381" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40103 } }
I20260620 23:09:45.926376 30555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:09:45.928257 30562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18426c99f3d64fe2b96324d47fcf5381. Latest consensus state: current_term: 1 leader_uuid: "18426c99f3d64fe2b96324d47fcf5381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18426c99f3d64fe2b96324d47fcf5381" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40103 } } }
I20260620 23:09:45.928699 30562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:45.930270 30566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:09:45.928280 30561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18426c99f3d64fe2b96324d47fcf5381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18426c99f3d64fe2b96324d47fcf5381" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 40103 } } }
I20260620 23:09:45.930655 30561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381 [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:45.935218 30566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:09:45.944105 30566 catalog_manager.cc:1347] Generated new cluster ID: 6a583ad2d205415b8ab5c2d60eb9353a
I20260620 23:09:45.944222 30566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:09:45.960862 30566 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:09:45.962227 30566 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:09:45.974406 30566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381: Generated new TSK 0
I20260620 23:09:45.975335 30566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:09:46.046137 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40103
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:46.368191 30581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:46.368502 30581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:46.368639 30581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:46.376436 30581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:46.376570 30581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:46.376762 30581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:09:46.386952 30581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:46.388365 30581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:46.389744 30581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:46.402741 30589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:46.402832 30586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:46.402807 30587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:46.403606 30581 server_base.cc:1034] running on GCE node
I20260620 23:09:46.404222 30581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:46.405251 30581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:46.406497 30581 hybrid_clock.cc:648] HybridClock initialized: now 1781996986406413 us; error 76 us; skew 500 ppm
I20260620 23:09:46.406883 30581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:46 dist-test-slave-7g8g krb5kdc[30475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996986, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:46.410058 30581 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:09:46.411482 30581 webserver.cc:460] Webserver started at http://127.12.2.65:34787/ using document root <none> and password file <none>
I20260620 23:09:46.412153 30581 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:46.412268 30581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:46.412562 30581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:46.415539 30581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "4d3b983ed48c45a49b3ac501e97532c3"
format_stamp: "Formatted at 2026-06-20 23:09:46 on dist-test-slave-7g8g"
server_key: "32035b93925060a75fafc50659810d91"
server_key_iv: "3d2dc89c485a3e9412e5ddc7fd0b5b33"
server_key_version: "encryptionkey@0"
I20260620 23:09:46.416276 30581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "4d3b983ed48c45a49b3ac501e97532c3"
format_stamp: "Formatted at 2026-06-20 23:09:46 on dist-test-slave-7g8g"
server_key: "32035b93925060a75fafc50659810d91"
server_key_iv: "3d2dc89c485a3e9412e5ddc7fd0b5b33"
server_key_version: "encryptionkey@0"
I20260620 23:09:46.421396 30581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:09:46.424753 30596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:46.426091 30581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:09:46.426241 30581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "4d3b983ed48c45a49b3ac501e97532c3"
format_stamp: "Formatted at 2026-06-20 23:09:46 on dist-test-slave-7g8g"
server_key: "32035b93925060a75fafc50659810d91"
server_key_iv: "3d2dc89c485a3e9412e5ddc7fd0b5b33"
server_key_version: "encryptionkey@0"
I20260620 23:09:46.426431 30581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:46.441341 30581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:46.445250 30581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:46.445508 30581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:46.446431 30581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:46.447755 30581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:46.447832 30581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:46.447954 30581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:46.448009 30581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:46.477098 30581 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:34515
I20260620 23:09:46.477160 30708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:34515 every 8 connection(s)
I20260620 23:09:46.478314 30581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:09:46.483280 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 30581
I20260620 23:09:46.483428 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:09:46.483847 12297 external_mini_cluster.cc:1428] Setting key 182971b9b87a4a8d7585ef2c73ab27bb
Jun 20 23:09:46 dist-test-slave-7g8g krb5kdc[30475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996986, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:46.497754 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:46.480680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:41101 (local address 127.12.2.126:40103)
0620 23:09:46.481094 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:46.481100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:46.482120 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 23:09:46.483948 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:46.483954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:46.484040 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:46.484136 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:46.485789 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:46.486480 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:46.488052 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:46.488456 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:46.488563 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:46.492100 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:46.492106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:46.492110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:46.492151 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:46.494716 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:46.495585 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:46.495589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:46.495591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:46.495712 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:46.496123 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:46.496126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:46.496128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:46.496380 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:46.496498 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:46.497269 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 23:09:46.497503 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":184,"threads_started":1}
I20260620 23:09:46.499037 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:46.481254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:40103 (local address 127.12.2.65:41101)
0620 23:09:46.481965 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:46.482003 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:46.483686 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:46.484373 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:46.484383 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:46.484966 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:46.485629 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:46.485648 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:46.486631 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:46.486635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:46.487903 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:46.487926 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:46.488665 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:46.488672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:46.488908 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:46.489532 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:09:46.489554 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:46.491869 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:46.494852 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:46.494861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:46.494879 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:46.495473 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:46.495800 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:46.495804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:46.495807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:46.496036 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:46.496485 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:46.496492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:46.496909 (+ 417us) client_negotiation.cc:769] Sending connection context
0620 23:09:46.497180 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 23:09:46.497708 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":266,"threads_started":1}
I20260620 23:09:46.500579 30709 heartbeater.cc:347] Connected to a master server at 127.12.2.126:40103
I20260620 23:09:46.500892 30709 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:46.501725 30709 heartbeater.cc:511] Master 127.12.2.126:40103 requested a full tablet report, sending...
I20260620 23:09:46.503867 30520 ts_manager.cc:194] Registered new tserver with Master: 4d3b983ed48c45a49b3ac501e97532c3 (127.12.2.65:34515)
I20260620 23:09:46.505363 30520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:41101
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:09:46.540709 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40103
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:46.859807 30715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:46.860108 30715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:46.860260 30715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:46.867442 30715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:46.867568 30715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:46.867730 30715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:09:46.877545 30715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:46.878839 30715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:46.880142 30715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:46.892621 30720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:46.892664 30721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:46.893141 30723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:46.893828 30715 server_base.cc:1034] running on GCE node
I20260620 23:09:46.894409 30715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:46.895359 30715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:46.896564 30715 hybrid_clock.cc:648] HybridClock initialized: now 1781996986896510 us; error 49 us; skew 500 ppm
I20260620 23:09:46.896849 30715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:46 dist-test-slave-7g8g krb5kdc[30475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996986, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:46.899870 30715 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:09:46.901365 30715 webserver.cc:460] Webserver started at http://127.12.2.66:37399/ using document root <none> and password file <none>
I20260620 23:09:46.901993 30715 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:46.902104 30715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:46.902379 30715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:46.905154 30715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "eb41ed21acf34066988510a2b4cb6326"
format_stamp: "Formatted at 2026-06-20 23:09:46 on dist-test-slave-7g8g"
server_key: "cbce9fa17f44636d8cb5c29bef9a3ed3"
server_key_iv: "0fc7e97d75099eeb7a7ea746fce95424"
server_key_version: "encryptionkey@0"
I20260620 23:09:46.905856 30715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "eb41ed21acf34066988510a2b4cb6326"
format_stamp: "Formatted at 2026-06-20 23:09:46 on dist-test-slave-7g8g"
server_key: "cbce9fa17f44636d8cb5c29bef9a3ed3"
server_key_iv: "0fc7e97d75099eeb7a7ea746fce95424"
server_key_version: "encryptionkey@0"
I20260620 23:09:46.910599 30715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260620 23:09:46.913900 30730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:46.915170 30715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:09:46.915315 30715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "eb41ed21acf34066988510a2b4cb6326"
format_stamp: "Formatted at 2026-06-20 23:09:46 on dist-test-slave-7g8g"
server_key: "cbce9fa17f44636d8cb5c29bef9a3ed3"
server_key_iv: "0fc7e97d75099eeb7a7ea746fce95424"
server_key_version: "encryptionkey@0"
I20260620 23:09:46.915490 30715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:46.931908 30715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:46.935942 30715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:46.936203 30715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:46.937108 30715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:46.938441 30715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:46.938515 30715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:46.938630 30715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:46.938685 30715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:46.967717 30715 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:41739
I20260620 23:09:46.967753 30842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:41739 every 8 connection(s)
I20260620 23:09:46.968953 30715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:09:46.978428 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 30715
I20260620 23:09:46.978596 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:09:46.979030 12297 external_mini_cluster.cc:1428] Setting key e1e4b58b556e4947a69fe8b1c5b014f9
Jun 20 23:09:46 dist-test-slave-7g8g krb5kdc[30475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996986, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:46.988668 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:46.971554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:47135 (local address 127.12.2.126:40103)
0620 23:09:46.971727 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:46.971731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:46.972838 (+ 1107us) server_negotiation.cc:407] Connection header received
0620 23:09:46.974599 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:46.974605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:46.974703 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:46.974791 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:46.976270 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:46.976987 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:46.978456 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:46.978925 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:46.979063 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:46.982891 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:46.982896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:46.982901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:46.982959 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:46.985700 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:46.986476 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:46.986482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:46.986485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:46.986630 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:46.987014 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:46.987018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:46.987022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:46.987347 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:46.987491 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:46.988123 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 23:09:46.988333 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:09:46.989508 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:46.971990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:40103 (local address 127.12.2.66:47135)
0620 23:09:46.972681 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:46.972718 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:46.974332 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:46.975070 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:46.975079 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:46.975575 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:46.976114 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:46.976130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:46.977127 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:46.977130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:46.978322 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:46.978333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:46.979155 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:46.979162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:46.979371 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:46.980027 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:09:46.980064 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:46.982652 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:46.985833 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:46.985840 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:46.985853 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:46.986369 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:46.986731 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:46.986734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:46.986736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:46.986908 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:46.987456 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:46.987462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:46.987790 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 23:09:46.988013 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 23:09:46.988394 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":267,"threads_started":1}
I20260620 23:09:46.991065 30843 heartbeater.cc:347] Connected to a master server at 127.12.2.126:40103
I20260620 23:09:46.991349 30843 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:46.992064 30843 heartbeater.cc:511] Master 127.12.2.126:40103 requested a full tablet report, sending...
I20260620 23:09:46.993515 30520 ts_manager.cc:194] Registered new tserver with Master: eb41ed21acf34066988510a2b4cb6326 (127.12.2.66:41739)
I20260620 23:09:46.994285 30520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:47135
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:09:47.033841 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40103
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:47.349207 30848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:47.349474 30848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:47.349604 30848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:47.356688 30848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:47.356803 30848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:47.356962 30848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:09:47.366313 30848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:40103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:47.367561 30848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:47.368829 30848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:47.380936 30856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:47.381095 30854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:47.381237 30853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:47.382280 30848 server_base.cc:1034] running on GCE node
I20260620 23:09:47.382825 30848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:47.383750 30848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:47.384969 30848 hybrid_clock.cc:648] HybridClock initialized: now 1781996987384880 us; error 99 us; skew 500 ppm
I20260620 23:09:47.385275 30848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:47 dist-test-slave-7g8g krb5kdc[30475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996987, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:47.389066 30848 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:09:47.390897 30848 webserver.cc:460] Webserver started at http://127.12.2.67:35523/ using document root <none> and password file <none>
I20260620 23:09:47.391791 30848 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:47.391901 30848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:47.392241 30848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:47.395885 30848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc"
format_stamp: "Formatted at 2026-06-20 23:09:47 on dist-test-slave-7g8g"
server_key: "55a415bcf0198e39197d52e51c3faaf9"
server_key_iv: "4c3946d5da890807559032367e0b8966"
server_key_version: "encryptionkey@0"
I20260620 23:09:47.396847 30848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc"
format_stamp: "Formatted at 2026-06-20 23:09:47 on dist-test-slave-7g8g"
server_key: "55a415bcf0198e39197d52e51c3faaf9"
server_key_iv: "4c3946d5da890807559032367e0b8966"
server_key_version: "encryptionkey@0"
I20260620 23:09:47.402882 30848 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 23:09:47.406786 30863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:47.408435 30848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:09:47.408608 30848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc"
format_stamp: "Formatted at 2026-06-20 23:09:47 on dist-test-slave-7g8g"
server_key: "55a415bcf0198e39197d52e51c3faaf9"
server_key_iv: "4c3946d5da890807559032367e0b8966"
server_key_version: "encryptionkey@0"
I20260620 23:09:47.408823 30848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:47.444525 30848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:47.448567 30848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:47.448820 30848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:47.449754 30848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:47.451076 30848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:47.451145 30848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:47.451261 30848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:47.451309 30848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:47.483196 30848 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:39617
I20260620 23:09:47.483242 30975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:39617 every 8 connection(s)
I20260620 23:09:47.484390 30848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:09:47.486398 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 30848
I20260620 23:09:47.486563 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:09:47.487113 12297 external_mini_cluster.cc:1428] Setting key 7f8e3f96da33a413335778cf361580d3
Jun 20 23:09:47 dist-test-slave-7g8g krb5kdc[30475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996987, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:47.503120 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.486858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:46367 (local address 127.12.2.126:40103)
0620 23:09:47.487103 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:47.487109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:47.488220 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 23:09:47.490484 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:47.490490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:47.490568 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:47.490648 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:47.492099 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.492828 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.494320 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.494705 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.494819 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.497965 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:47.497969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:47.497972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:47.498004 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:47.500280 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:47.500993 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:47.500997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:47.501000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:47.501170 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:47.501561 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:47.501566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:47.501569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:47.501858 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:47.501999 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:47.502657 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 23:09:47.502814 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 23:09:47.503953 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.487309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:40103 (local address 127.12.2.67:46367)
0620 23:09:47.488087 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:47.488134 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:47.490231 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:47.490879 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:47.490887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:47.491405 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:47.491939 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.491957 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.492984 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.492987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.494184 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.494197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.494912 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.494920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.495174 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.495737 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:09:47.495761 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:47.497768 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:47.500411 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:47.500419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:47.500435 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:47.500871 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:47.501269 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:47.501273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:47.501275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:47.501459 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:47.501989 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:47.501996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:47.502332 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 23:09:47.502560 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 23:09:47.502912 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":361,"threads_started":1}
I20260620 23:09:47.505113 30976 heartbeater.cc:347] Connected to a master server at 127.12.2.126:40103
I20260620 23:09:47.505362 30976 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:47.505987 30976 heartbeater.cc:511] Master 127.12.2.126:40103 requested a full tablet report, sending...
I20260620 23:09:47.507252 30520 ts_manager.cc:194] Registered new tserver with Master: ddaa81332f8e4c9baf966ac1dec0e2fc (127.12.2.67:39617)
I20260620 23:09:47.507993 30520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:46367
I20260620 23:09:47.508800 30709 heartbeater.cc:503] Master 127.12.2.126:40103 was elected leader, sending a full tablet report...
I20260620 23:09:47.514061 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:09:47.530090 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.517957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.12.2.126:40103)
0620 23:09:47.518109 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:47.518115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:47.518517 (+ 402us) server_negotiation.cc:407] Connection header received
0620 23:09:47.518660 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:47.518665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:47.518724 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:47.518802 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:47.520156 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.520851 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.522118 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.522523 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.522634 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.525074 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:47.525079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:47.525082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:47.525111 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:47.527486 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:47.528225 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:47.528229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:47.528231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:47.528345 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:47.528752 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:47.528756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:47.528758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:47.529046 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:47.529301 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:47.529656 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 23:09:47.529823 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:09:47.534000 30520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:09:47.540720 30520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:09:47.572104 30988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.552313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:34515 (local address 127.0.0.1:56500)
0620 23:09:47.553139 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:47.553185 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:47.553352 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:47.564451 (+ 11099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:47.564458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:47.564492 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:47.565366 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.565381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.567104 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.567108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.569452 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.569467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.571104 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.571110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.571287 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.571336 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:09:47.571480 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 23:09:47.571667 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":1,"thread_start_us":12756,"threads_started":1}
I20260620 23:09:47.572431 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.552576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.12.2.65:34515)
0620 23:09:47.563902 (+ 11326us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:47.563908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:47.563932 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:09:47.564016 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:47.564022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:47.564234 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:47.564416 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:47.565614 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.566983 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.569616 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.570876 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.571056 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.571286 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:47.571527 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 23:09:47.571687 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11179,"thread_start_us":11034,"threads_started":1}
I20260620 23:09:47.573077 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.552371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:39617 (local address 127.0.0.1:42968)
0620 23:09:47.565211 (+ 12840us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:47.565234 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:47.565382 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:47.566334 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:47.566338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:47.566375 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:47.566825 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.566840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.568188 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.568193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.570187 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.570201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.572053 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.572060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.572210 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.572240 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:09:47.572624 (+ 384us) client_negotiation.cc:240] Negotiation successful
0620 23:09:47.572747 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12702,"thread_start_us":637,"threads_started":1}
I20260620 23:09:47.573490 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.553108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.12.2.66:41739)
0620 23:09:47.563438 (+ 10330us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:47.563448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:47.565566 (+ 2118us) server_negotiation.cc:407] Connection header received
0620 23:09:47.565853 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:47.565861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:47.566087 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:47.566243 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:47.567073 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.568615 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.570696 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.571999 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.572117 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.572284 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:47.572856 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 23:09:47.573009 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10173,"thread_start_us":10043,"threads_started":1}
I20260620 23:09:47.573784 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.552854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.12.2.67:39617)
0620 23:09:47.564499 (+ 11645us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:47.564508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:47.565527 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 23:09:47.565636 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:47.565643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:47.565882 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:47.566069 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:47.567044 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.568059 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.570350 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.571588 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.571693 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.571906 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:47.573397 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0620 23:09:47.573519 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11452,"thread_start_us":11821,"threads_started":1}
I20260620 23:09:47.574201 30989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.552803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:41739 (local address 127.0.0.1:41374)
0620 23:09:47.565464 (+ 12661us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:47.565482 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:47.565669 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:47.566428 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:47.566432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:47.566451 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:47.566882 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.566895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.568782 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.568791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.570545 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.570558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.572431 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.572436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.572677 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.572713 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:09:47.572792 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:09:47.572916 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12516,"mutex_wait_us":29,"thread_start_us":12107,"threads_started":1}
I20260620 23:09:47.576980 30644 tablet_service.cc:1467] Processing CreateTablet for tablet 69273945949d44fbb7ad924cde7d969f (DEFAULT_TABLE table=test-table [id=3bb8e3e0fda94475940f8cabbed42bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:47.577162 30911 tablet_service.cc:1467] Processing CreateTablet for tablet 69273945949d44fbb7ad924cde7d969f (DEFAULT_TABLE table=test-table [id=3bb8e3e0fda94475940f8cabbed42bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:47.577346 30778 tablet_service.cc:1467] Processing CreateTablet for tablet 69273945949d44fbb7ad924cde7d969f (DEFAULT_TABLE table=test-table [id=3bb8e3e0fda94475940f8cabbed42bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:47.578388 30911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69273945949d44fbb7ad924cde7d969f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:47.578451 30778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69273945949d44fbb7ad924cde7d969f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:47.578450 30644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69273945949d44fbb7ad924cde7d969f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:47.589107 30994 tablet_bootstrap.cc:492] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc: Bootstrap starting.
I20260620 23:09:47.592064 30995 tablet_bootstrap.cc:492] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326: Bootstrap starting.
I20260620 23:09:47.592397 30994 tablet_bootstrap.cc:654] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:47.593720 30994 log.cc:824] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:47.594527 30996 tablet_bootstrap.cc:492] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3: Bootstrap starting.
I20260620 23:09:47.595438 30995 tablet_bootstrap.cc:654] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:47.596660 30995 log.cc:824] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:47.597465 30996 tablet_bootstrap.cc:654] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:47.598448 30996 log.cc:824] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:47.599731 30995 tablet_bootstrap.cc:492] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326: No bootstrap required, opened a new log
I20260620 23:09:47.599994 30995 ts_tablet_manager.cc:1397] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260620 23:09:47.601053 30994 tablet_bootstrap.cc:492] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc: No bootstrap required, opened a new log
I20260620 23:09:47.601260 30996 tablet_bootstrap.cc:492] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3: No bootstrap required, opened a new log
I20260620 23:09:47.601274 30994 ts_tablet_manager.cc:1397] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260620 23:09:47.601497 30996 ts_tablet_manager.cc:1397] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 23:09:47.608422 30994 raft_consensus.cc:348] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b983ed48c45a49b3ac501e97532c3" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 34515 } } peers { permanent_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 39617 } } peers { permanent_uuid: "eb41ed21acf34066988510a2b4cb6326" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 41739 } }
I20260620 23:09:47.608777 30994 raft_consensus.cc:374] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:47.608878 30994 raft_consensus.cc:729] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaa81332f8e4c9baf966ac1dec0e2fc, State: Initialized, Role: FOLLOWER
I20260620 23:09:47.609449 30994 consensus_queue.cc:260] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b983ed48c45a49b3ac501e97532c3" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 34515 } } peers { permanent_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 39617 } } peers { permanent_uuid: "eb41ed21acf34066988510a2b4cb6326" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 41739 } }
I20260620 23:09:47.609985 30995 raft_consensus.cc:348] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b983ed48c45a49b3ac501e97532c3" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 34515 } } peers { permanent_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 39617 } } peers { permanent_uuid: "eb41ed21acf34066988510a2b4cb6326" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 41739 } }
I20260620 23:09:47.610394 30995 raft_consensus.cc:374] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:47.610466 30976 heartbeater.cc:503] Master 127.12.2.126:40103 was elected leader, sending a full tablet report...
I20260620 23:09:47.610473 30995 raft_consensus.cc:729] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb41ed21acf34066988510a2b4cb6326, State: Initialized, Role: FOLLOWER
I20260620 23:09:47.611171 30994 ts_tablet_manager.cc:1428] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc: Time spent starting tablet: real 0.010s user 0.005s sys 0.006s
I20260620 23:09:47.611124 30995 consensus_queue.cc:260] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b983ed48c45a49b3ac501e97532c3" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 34515 } } peers { permanent_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 39617 } } peers { permanent_uuid: "eb41ed21acf34066988510a2b4cb6326" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 41739 } }
I20260620 23:09:47.611755 30996 raft_consensus.cc:348] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b983ed48c45a49b3ac501e97532c3" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 34515 } } peers { permanent_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 39617 } } peers { permanent_uuid: "eb41ed21acf34066988510a2b4cb6326" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 41739 } }
I20260620 23:09:47.612118 30996 raft_consensus.cc:374] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:47.612200 30996 raft_consensus.cc:729] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3b983ed48c45a49b3ac501e97532c3, State: Initialized, Role: FOLLOWER
I20260620 23:09:47.612229 30843 heartbeater.cc:503] Master 127.12.2.126:40103 was elected leader, sending a full tablet report...
I20260620 23:09:47.612749 30995 ts_tablet_manager.cc:1428] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260620 23:09:47.612821 30996 consensus_queue.cc:260] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b983ed48c45a49b3ac501e97532c3" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 34515 } } peers { permanent_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 39617 } } peers { permanent_uuid: "eb41ed21acf34066988510a2b4cb6326" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 41739 } }
I20260620 23:09:47.615597 30996 ts_tablet_manager.cc:1428] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260620 23:09:47.635367 31001 raft_consensus.cc:482] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:09:47.635634 31001 raft_consensus.cc:504] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b983ed48c45a49b3ac501e97532c3" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 34515 } } peers { permanent_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 39617 } } peers { permanent_uuid: "eb41ed21acf34066988510a2b4cb6326" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 41739 } }
I20260620 23:09:47.637290 31001 leader_election.cc:288] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3b983ed48c45a49b3ac501e97532c3 (127.12.2.65:34515), ddaa81332f8e4c9baf966ac1dec0e2fc (127.12.2.67:39617)
I20260620 23:09:47.643532 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.637605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:59161 (local address 127.12.2.65:34515)
0620 23:09:47.637796 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:47.637803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:47.638460 (+ 657us) server_negotiation.cc:407] Connection header received
0620 23:09:47.638627 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:47.638631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:47.638686 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:47.638757 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:47.639378 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.640045 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.641809 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.642613 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.642726 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.642827 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:47.643181 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 23:09:47.643296 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:09:47.643561 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.637679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:58621 (local address 127.12.2.67:39617)
0620 23:09:47.637865 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:47.637870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:47.638534 (+ 664us) server_negotiation.cc:407] Connection header received
0620 23:09:47.638686 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:47.638690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:47.638747 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:47.638818 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:47.639455 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.640135 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.641847 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.642633 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.642734 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.642832 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:47.643205 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 23:09:47.643324 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:09:47.643812 31004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.637673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:39617 (local address 127.12.2.66:58621)
0620 23:09:47.638431 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:47.638446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:47.638562 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:47.638842 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:47.638846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:47.638887 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:47.639301 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.639310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.640266 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.640273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.641706 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.641718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.642845 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.642852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.643064 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.643087 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:09:47.643165 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:09:47.643288 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":41,"thread_start_us":426,"threads_started":1}
I20260620 23:09:47.644054 31003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.637499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:34515 (local address 127.12.2.66:59161)
0620 23:09:47.638356 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:47.638372 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:47.638506 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:47.638783 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:47.638788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:47.638809 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:47.639233 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.639242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.640157 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.640162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.641673 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:47.641684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.642822 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.642832 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:47.643022 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.643052 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:09:47.643137 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:09:47.643259 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":51,"thread_start_us":503,"threads_started":1}
I20260620 23:09:47.644769 30931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69273945949d44fbb7ad924cde7d969f" candidate_uuid: "eb41ed21acf34066988510a2b4cb6326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" is_pre_election: true
I20260620 23:09:47.644994 30664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69273945949d44fbb7ad924cde7d969f" candidate_uuid: "eb41ed21acf34066988510a2b4cb6326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3b983ed48c45a49b3ac501e97532c3" is_pre_election: true
I20260620 23:09:47.645300 30931 raft_consensus.cc:2454] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb41ed21acf34066988510a2b4cb6326 in term 0.
I20260620 23:09:47.645505 30664 raft_consensus.cc:2454] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb41ed21acf34066988510a2b4cb6326 in term 0.
I20260620 23:09:47.646009 30733 leader_election.cc:302] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddaa81332f8e4c9baf966ac1dec0e2fc, eb41ed21acf34066988510a2b4cb6326; no voters:
I20260620 23:09:47.646386 31001 raft_consensus.cc:2781] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:09:47.646522 31001 raft_consensus.cc:482] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:09:47.646622 31001 raft_consensus.cc:3037] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:47.648859 31001 raft_consensus.cc:504] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b983ed48c45a49b3ac501e97532c3" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 34515 } } peers { permanent_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 39617 } } peers { permanent_uuid: "eb41ed21acf34066988510a2b4cb6326" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 41739 } }
I20260620 23:09:47.649426 31001 leader_election.cc:288] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3b983ed48c45a49b3ac501e97532c3 (127.12.2.65:34515), ddaa81332f8e4c9baf966ac1dec0e2fc (127.12.2.67:39617)
I20260620 23:09:47.649785 30664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69273945949d44fbb7ad924cde7d969f" candidate_uuid: "eb41ed21acf34066988510a2b4cb6326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3b983ed48c45a49b3ac501e97532c3"
I20260620 23:09:47.649871 30931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69273945949d44fbb7ad924cde7d969f" candidate_uuid: "eb41ed21acf34066988510a2b4cb6326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc"
I20260620 23:09:47.649953 30664 raft_consensus.cc:3037] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:47.650074 30931 raft_consensus.cc:3037] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:47.652120 30664 raft_consensus.cc:2454] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb41ed21acf34066988510a2b4cb6326 in term 1.
I20260620 23:09:47.652453 30931 raft_consensus.cc:2454] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb41ed21acf34066988510a2b4cb6326 in term 1.
I20260620 23:09:47.652513 30731 leader_election.cc:302] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3b983ed48c45a49b3ac501e97532c3, eb41ed21acf34066988510a2b4cb6326; no voters:
I20260620 23:09:47.652817 31001 raft_consensus.cc:2781] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:47.653244 31001 raft_consensus.cc:686] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 1 LEADER]: Becoming Leader. State: Replica: eb41ed21acf34066988510a2b4cb6326, State: Running, Role: LEADER
I20260620 23:09:47.653673 31001 consensus_queue.cc:237] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b983ed48c45a49b3ac501e97532c3" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 34515 } } peers { permanent_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 39617 } } peers { permanent_uuid: "eb41ed21acf34066988510a2b4cb6326" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 41739 } }
I20260620 23:09:47.658159 30520 catalog_manager.cc:5495] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb41ed21acf34066988510a2b4cb6326 (127.12.2.66). New cstate: current_term: 1 leader_uuid: "eb41ed21acf34066988510a2b4cb6326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b983ed48c45a49b3ac501e97532c3" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 34515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb41ed21acf34066988510a2b4cb6326" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 41739 } health_report { overall_health: HEALTHY } } }
I20260620 23:09:47.675779 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.670773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.12.2.66:41739)
0620 23:09:47.670997 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:47.671002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:47.671039 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:09:47.671212 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:47.671215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:47.671290 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:47.671410 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:09:47.671717 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.672436 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.673694 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.674091 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.674183 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.674642 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:09:47.675232 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:09:47.675386 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:47.675448 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 23:09:47.675491 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:09:47.683501 30931 raft_consensus.cc:1261] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc [term 1 FOLLOWER]: Refusing update from remote peer eb41ed21acf34066988510a2b4cb6326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:47.683549 30664 raft_consensus.cc:1261] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3 [term 1 FOLLOWER]: Refusing update from remote peer eb41ed21acf34066988510a2b4cb6326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:47.684463 31005 consensus_queue.cc:1035] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddaa81332f8e4c9baf966ac1dec0e2fc" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:09:47.684787 31001 consensus_queue.cc:1035] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3b983ed48c45a49b3ac501e97532c3" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:09:47.711273 30519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:47.711607 30519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:47.715653 30519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18426c99f3d64fe2b96324d47fcf5381: Sending DeleteTablet for 3 replicas of tablet 69273945949d44fbb7ad924cde7d969f
I20260620 23:09:47.716877 30644 tablet_service.cc:1514] Processing DeleteTablet for tablet 69273945949d44fbb7ad924cde7d969f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:47 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:56500
I20260620 23:09:47.717010 30778 tablet_service.cc:1514] Processing DeleteTablet for tablet 69273945949d44fbb7ad924cde7d969f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:47 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:41374
I20260620 23:09:47.717648 30911 tablet_service.cc:1514] Processing DeleteTablet for tablet 69273945949d44fbb7ad924cde7d969f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:47 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:42968
I20260620 23:09:47.717800 31018 tablet_replica.cc:330] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3: stopping tablet replica
I20260620 23:09:47.718223 31018 raft_consensus.cc:2229] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:47.718830 31018 raft_consensus.cc:2258] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:47.719550 31019 tablet_replica.cc:330] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326: stopping tablet replica
I20260620 23:09:47.719938 31019 raft_consensus.cc:2229] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:09:47.720584 31019 raft_consensus.cc:2258] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:47.721741 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 30581
I20260620 23:09:47.722649 31019 ts_tablet_manager.cc:1905] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:47.726615 31018 ts_tablet_manager.cc:1905] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:47.729004 31020 tablet_replica.cc:330] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc: stopping tablet replica
I20260620 23:09:47.729485 31020 raft_consensus.cc:2229] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:47.729998 31020 raft_consensus.cc:2258] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:47.732319 31020 ts_tablet_manager.cc:1905] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:47.734546 31018 ts_tablet_manager.cc:1918] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:47.734710 31018 log.cc:1196] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/69273945949d44fbb7ad924cde7d969f
I20260620 23:09:47.738874 31020 ts_tablet_manager.cc:1918] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:47.739045 31020 log.cc:1196] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/69273945949d44fbb7ad924cde7d969f
I20260620 23:09:47.739538 31020 ts_tablet_manager.cc:1939] T 69273945949d44fbb7ad924cde7d969f P ddaa81332f8e4c9baf966ac1dec0e2fc: Deleting consensus metadata
I20260620 23:09:47.740082 31019 ts_tablet_manager.cc:1918] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:47.740247 31019 log.cc:1196] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/69273945949d44fbb7ad924cde7d969f
I20260620 23:09:47.740790 31019 ts_tablet_manager.cc:1939] T 69273945949d44fbb7ad924cde7d969f P eb41ed21acf34066988510a2b4cb6326: Deleting consensus metadata
I20260620 23:09:47.741763 30506 catalog_manager.cc:4841] TS ddaa81332f8e4c9baf966ac1dec0e2fc (127.12.2.67:39617): tablet 69273945949d44fbb7ad924cde7d969f (table test-table [id=3bb8e3e0fda94475940f8cabbed42bf9]) successfully deleted
I20260620 23:09:47.742431 30505 catalog_manager.cc:4841] TS eb41ed21acf34066988510a2b4cb6326 (127.12.2.66:41739): tablet 69273945949d44fbb7ad924cde7d969f (table test-table [id=3bb8e3e0fda94475940f8cabbed42bf9]) successfully deleted
I20260620 23:09:47.742697 31018 ts_tablet_manager.cc:1939] T 69273945949d44fbb7ad924cde7d969f P 4d3b983ed48c45a49b3ac501e97532c3: Deleting consensus metadata
I20260620 23:09:47.744331 30504 catalog_manager.cc:4841] TS 4d3b983ed48c45a49b3ac501e97532c3 (127.12.2.65:34515): tablet 69273945949d44fbb7ad924cde7d969f (table test-table [id=3bb8e3e0fda94475940f8cabbed42bf9]) successfully deleted
Jun 20 23:09:47 dist-test-slave-7g8g krb5kdc[30475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:09:47.753783 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.722859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.12.2.65:34515)
0620 23:09:47.723056 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:47.723062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:47.734309 (+ 11247us) server_negotiation.cc:407] Connection header received
0620 23:09:47.734904 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:47.734911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:47.735016 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:47.735088 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:47.736807 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.737903 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.739954 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.740521 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.740628 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.748449 (+ 7821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:47.748460 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:47.748474 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:47.748511 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:47.751196 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:47.751831 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:47.751841 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:47.751843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:47.751988 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:47.752339 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:47.752342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:47.752344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:47.752681 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:47.752980 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:47.753265 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 23:09:47.753467 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:09:47.754452 30704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:47.842248 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 30581
I20260620 23:09:47.856757 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 30715
Jun 20 23:09:47 dist-test-slave-7g8g krb5kdc[30475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:09:47.872924 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.857445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.12.2.66:41739)
0620 23:09:47.857597 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:47.857602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:47.857803 (+ 201us) server_negotiation.cc:407] Connection header received
0620 23:09:47.858010 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:47.858014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:47.858081 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:47.858169 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:47.859498 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.860286 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.861583 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:47.862014 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:47.862120 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:47.866284 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:47.866302 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:47.866323 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:47.866379 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:47.870127 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:47.870774 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:47.870783 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:47.870787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:47.870970 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:47.871338 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:47.871343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:47.871346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:47.871777 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:47.872185 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:47.872333 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 23:09:47.872558 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:09:47.873672 30838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:47.983477 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 30715
I20260620 23:09:47.998088 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 30848
Jun 20 23:09:48 dist-test-slave-7g8g krb5kdc[30475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:09:48.012959 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:47.998819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.12.2.67:39617)
0620 23:09:47.999005 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:47.999009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:47.999027 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:09:47.999157 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:47.999161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:47.999231 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:47.999312 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:48.000555 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:48.001305 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:48.002711 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:48.003255 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:48.003343 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:48.007658 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:48.007673 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:48.007689 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:48.007722 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:48.010109 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:48.010798 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:48.010805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:48.010807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:48.010959 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:48.011381 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:48.011386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:48.011389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:48.011842 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:48.012115 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:48.012499 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 23:09:48.012675 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:09:48.014305 30971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:48.101076 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 30848
I20260620 23:09:48.114297 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 30488
I20260620 23:09:48.115375 30550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:48.222800 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 30488
2026-06-20T23:09:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3993 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T23:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:09:48Z Disabled control of system clock
I20260620 23:09:48.290364 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:36613
--webserver_interface=127.12.2.126
--webserver_port=0
--builtin_ntp_servers=127.12.2.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:09:48.602245 31029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:48.602502 31029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:48.602588 31029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:48.609555 31029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:09:48.609649 31029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:48.609691 31029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:48.609745 31029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:09:48.609786 31029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:09:48.618891 31029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:36613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:48.620113 31029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:48.621852 31029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:48.633268 31037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:48.633337 31034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:48.633345 31035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:48.633669 31029 server_base.cc:1034] running on GCE node
I20260620 23:09:48.634552 31029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:48.635914 31029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:48.637138 31029 hybrid_clock.cc:648] HybridClock initialized: now 1781996988637089 us; error 34 us; skew 500 ppm
I20260620 23:09:48.637455 31029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:48.639542 31029 webserver.cc:460] Webserver started at http://127.12.2.126:37085/ using document root <none> and password file <none>
I20260620 23:09:48.640169 31029 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:48.640265 31029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:48.640588 31029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:48.643420 31029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "2a02c07818e74029a83a2d427b2efb9a"
format_stamp: "Formatted at 2026-06-20 23:09:48 on dist-test-slave-7g8g"
server_key: "45c02ef7d1672233da46e67545574cb0"
server_key_iv: "2c431a0d4f72a3936b1044c62a955a3d"
server_key_version: "encryptionkey@0"
I20260620 23:09:48.644189 31029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "2a02c07818e74029a83a2d427b2efb9a"
format_stamp: "Formatted at 2026-06-20 23:09:48 on dist-test-slave-7g8g"
server_key: "45c02ef7d1672233da46e67545574cb0"
server_key_iv: "2c431a0d4f72a3936b1044c62a955a3d"
server_key_version: "encryptionkey@0"
I20260620 23:09:48.648998 31029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 23:09:48.652376 31043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:48.653666 31029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:09:48.653824 31029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "2a02c07818e74029a83a2d427b2efb9a"
format_stamp: "Formatted at 2026-06-20 23:09:48 on dist-test-slave-7g8g"
server_key: "45c02ef7d1672233da46e67545574cb0"
server_key_iv: "2c431a0d4f72a3936b1044c62a955a3d"
server_key_version: "encryptionkey@0"
I20260620 23:09:48.654062 31029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:48.672180 31029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:48.672976 31029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:48.673221 31029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:48.689595 31029 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:36613
I20260620 23:09:48.689604 31094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:36613 every 8 connection(s)
I20260620 23:09:48.690910 31029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:09:48.691599 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 31029
I20260620 23:09:48.691756 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:09:48.692291 12297 external_mini_cluster.cc:1428] Setting key 6fea04ddfb4d0819f06ccc5f6f7d669a
I20260620 23:09:48.695196 31095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:48.707262 31098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:48.694116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.12.2.126:36613)
0620 23:09:48.695196 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:48.695210 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:48.695259 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:09:48.697554 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:48.697581 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:48.698158 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:48.698610 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:48.699153 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:48.701131 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:48.702398 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:48.703418 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:48.703581 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:48.704569 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:48.704579 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:48.704596 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:48.705305 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:48.705409 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:48.705434 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:48.705542 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:48.705692 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 23:09:48.706024 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":310,"threads_started":1}
I20260620 23:09:48.708662 31095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a: Bootstrap starting.
I20260620 23:09:48.711794 31095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:48.712800 31095 log.cc:824] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:48.715304 31095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a: No bootstrap required, opened a new log
I20260620 23:09:48.722426 31095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02c07818e74029a83a2d427b2efb9a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36613 } }
I20260620 23:09:48.722817 31095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:48.722944 31095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a02c07818e74029a83a2d427b2efb9a, State: Initialized, Role: FOLLOWER
I20260620 23:09:48.723471 31095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02c07818e74029a83a2d427b2efb9a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36613 } }
I20260620 23:09:48.723693 31095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:09:48.723830 31095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:09:48.724004 31095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:48.726181 31095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02c07818e74029a83a2d427b2efb9a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36613 } }
I20260620 23:09:48.726614 31095 leader_election.cc:302] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a02c07818e74029a83a2d427b2efb9a; no voters:
I20260620 23:09:48.727146 31095 leader_election.cc:288] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:09:48.727252 31100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:48.727839 31100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [term 1 LEADER]: Becoming Leader. State: Replica: 2a02c07818e74029a83a2d427b2efb9a, State: Running, Role: LEADER
I20260620 23:09:48.728336 31100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02c07818e74029a83a2d427b2efb9a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36613 } }
I20260620 23:09:48.729072 31095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:09:48.730614 31101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a02c07818e74029a83a2d427b2efb9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02c07818e74029a83a2d427b2efb9a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36613 } } }
I20260620 23:09:48.731029 31101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:48.732755 31102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a02c07818e74029a83a2d427b2efb9a. Latest consensus state: current_term: 1 leader_uuid: "2a02c07818e74029a83a2d427b2efb9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02c07818e74029a83a2d427b2efb9a" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 36613 } } }
I20260620 23:09:48.733120 31102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:48.734046 31108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:09:48.739282 31108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:09:48.747196 31108 catalog_manager.cc:1347] Generated new cluster ID: 8ebb7091afc94c958dc3c803a6f8458e
I20260620 23:09:48.747303 31108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:09:48.771623 31108 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:09:48.773005 31108 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:09:48.780575 31108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a: Generated new TSK 0
I20260620 23:09:48.781569 31108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:09:48.789220 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36613
--builtin_ntp_servers=127.12.2.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:09:49.101913 31119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:49.102224 31119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:49.102370 31119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:49.109754 31119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:49.109891 31119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:49.110066 31119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:09:49.119438 31119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:49.120733 31119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:49.122570 31119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:49.135919 31127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:49.135990 31125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:49.135975 31124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:49.136826 31119 server_base.cc:1034] running on GCE node
I20260620 23:09:49.137482 31119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:49.138463 31119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:49.139654 31119 hybrid_clock.cc:648] HybridClock initialized: now 1781996989139604 us; error 47 us; skew 500 ppm
I20260620 23:09:49.139977 31119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:49.142055 31119 webserver.cc:460] Webserver started at http://127.12.2.65:37657/ using document root <none> and password file <none>
I20260620 23:09:49.142681 31119 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:49.142774 31119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:49.143095 31119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:49.145749 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "6bdcfac406ce4d71b57b78497eb55a22"
format_stamp: "Formatted at 2026-06-20 23:09:49 on dist-test-slave-7g8g"
server_key: "f2c7fb3a4fd06fe14515623af1f2b550"
server_key_iv: "ace4846f4d368f99d1fba2e5ed910fe8"
server_key_version: "encryptionkey@0"
I20260620 23:09:49.146474 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "6bdcfac406ce4d71b57b78497eb55a22"
format_stamp: "Formatted at 2026-06-20 23:09:49 on dist-test-slave-7g8g"
server_key: "f2c7fb3a4fd06fe14515623af1f2b550"
server_key_iv: "ace4846f4d368f99d1fba2e5ed910fe8"
server_key_version: "encryptionkey@0"
I20260620 23:09:49.151243 31119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 23:09:49.154484 31133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:49.155848 31119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:09:49.155987 31119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "6bdcfac406ce4d71b57b78497eb55a22"
format_stamp: "Formatted at 2026-06-20 23:09:49 on dist-test-slave-7g8g"
server_key: "f2c7fb3a4fd06fe14515623af1f2b550"
server_key_iv: "ace4846f4d368f99d1fba2e5ed910fe8"
server_key_version: "encryptionkey@0"
I20260620 23:09:49.156172 31119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:49.193482 31119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:49.194309 31119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:49.194563 31119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:49.195528 31119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:49.196947 31119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:49.197038 31119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:49.197145 31119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:49.197211 31119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:49.225178 31119 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:43279
I20260620 23:09:49.225229 31245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:43279 every 8 connection(s)
I20260620 23:09:49.226385 31119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:09:49.231474 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 31119
I20260620 23:09:49.231630 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:09:49.232084 12297 external_mini_cluster.cc:1428] Setting key d8edd11065fa45cb6f3f4810dbd89f7a
I20260620 23:09:49.234660 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36613
--builtin_ntp_servers=127.12.2.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:09:49.238389 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:49.228713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:33509 (local address 127.12.2.126:36613)
0620 23:09:49.229114 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:49.229120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:49.230348 (+ 1228us) server_negotiation.cc:407] Connection header received
0620 23:09:49.232152 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:49.232158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:49.232242 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:49.232354 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:49.233424 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:49.234297 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:49.235890 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:49.236275 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:49.236382 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:49.237630 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:49.237634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:49.237638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:49.237654 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:49.237703 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:49.237715 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:49.237787 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:49.237996 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 23:09:49.238088 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":173,"threads_started":1}
I20260620 23:09:49.239343 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:49.229324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36613 (local address 127.12.2.65:33509)
0620 23:09:49.230214 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:49.230253 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:49.231900 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:49.232540 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:49.232548 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:49.232642 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:49.233271 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:49.233289 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:49.234451 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:49.234455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:49.235751 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:49.235763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:49.236484 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:49.236491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:49.236716 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:49.237314 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:09:49.237317 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:49.237366 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:09:49.237449 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:49.237798 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:49.237810 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:49.237817 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:09:49.237971 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 23:09:49.238335 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":297,"threads_started":1}
I20260620 23:09:49.240803 31246 heartbeater.cc:347] Connected to a master server at 127.12.2.126:36613
I20260620 23:09:49.241097 31246 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:49.241789 31246 heartbeater.cc:511] Master 127.12.2.126:36613 requested a full tablet report, sending...
I20260620 23:09:49.243806 31060 ts_manager.cc:194] Registered new tserver with Master: 6bdcfac406ce4d71b57b78497eb55a22 (127.12.2.65:43279)
I20260620 23:09:49.245626 31060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.65:33509
W20260620 23:09:49.544549 31250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:49.544823 31250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:49.544906 31250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:49.552040 31250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:49.552137 31250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:49.552241 31250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:09:49.561339 31250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:49.562542 31250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:49.564234 31250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:49.581207 31258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:49.581238 31255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:49.581270 31256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:49.582062 31250 server_base.cc:1034] running on GCE node
I20260620 23:09:49.582636 31250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:49.583595 31250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:49.584781 31250 hybrid_clock.cc:648] HybridClock initialized: now 1781996989584729 us; error 42 us; skew 500 ppm
I20260620 23:09:49.585088 31250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:49.587229 31250 webserver.cc:460] Webserver started at http://127.12.2.66:36865/ using document root <none> and password file <none>
I20260620 23:09:49.587865 31250 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:49.587952 31250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:49.588240 31250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:49.590965 31250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "b4a4efbfe9c34901a3f214df789f998b"
format_stamp: "Formatted at 2026-06-20 23:09:49 on dist-test-slave-7g8g"
server_key: "a59a6b6b2c80de15efe65924ad88573d"
server_key_iv: "ca460637770b38731ae2639c496a3add"
server_key_version: "encryptionkey@0"
I20260620 23:09:49.591773 31250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "b4a4efbfe9c34901a3f214df789f998b"
format_stamp: "Formatted at 2026-06-20 23:09:49 on dist-test-slave-7g8g"
server_key: "a59a6b6b2c80de15efe65924ad88573d"
server_key_iv: "ca460637770b38731ae2639c496a3add"
server_key_version: "encryptionkey@0"
I20260620 23:09:49.596493 31250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:09:49.599807 31264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:49.601033 31250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:09:49.601152 31250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "b4a4efbfe9c34901a3f214df789f998b"
format_stamp: "Formatted at 2026-06-20 23:09:49 on dist-test-slave-7g8g"
server_key: "a59a6b6b2c80de15efe65924ad88573d"
server_key_iv: "ca460637770b38731ae2639c496a3add"
server_key_version: "encryptionkey@0"
I20260620 23:09:49.601311 31250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:49.612663 31250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:49.613361 31250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:49.613567 31250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:49.614356 31250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:49.615703 31250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:49.615783 31250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:49.615859 31250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:49.615911 31250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:49.644488 31250 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:44773
I20260620 23:09:49.644559 31376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:44773 every 8 connection(s)
I20260620 23:09:49.645694 31250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:09:49.648339 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 31250
I20260620 23:09:49.648494 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:09:49.648916 12297 external_mini_cluster.cc:1428] Setting key 8fb0414106aaf43fc5cc730e87a27d17
I20260620 23:09:49.651866 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36613
--builtin_ntp_servers=127.12.2.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:09:49.658885 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:49.648127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:42403 (local address 127.12.2.126:36613)
0620 23:09:49.648330 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:49.648336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:49.649771 (+ 1435us) server_negotiation.cc:407] Connection header received
0620 23:09:49.652013 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:49.652018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:49.652104 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:49.652200 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:49.653483 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:49.654251 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:49.656227 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:49.656631 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:49.656717 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:49.658137 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:49.658143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:49.658146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:49.658162 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:49.658193 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:49.658204 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:49.658291 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:49.658493 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 23:09:49.658591 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:09:49.660878 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:49.648616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36613 (local address 127.12.2.66:42403)
0620 23:09:49.649626 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:49.649666 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:49.651705 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:49.652433 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:49.652444 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:49.652570 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:49.653340 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:49.653360 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:49.654385 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:49.654391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:49.656115 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:49.656128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:49.656824 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:49.656831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:49.657146 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:49.657825 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:09:49.657827 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:49.657870 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:09:49.657944 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:49.658300 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:49.658311 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:49.658319 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:09:49.658486 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 23:09:49.659739 (+ 1253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":311,"threads_started":1}
I20260620 23:09:49.662151 31377 heartbeater.cc:347] Connected to a master server at 127.12.2.126:36613
I20260620 23:09:49.662475 31377 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:49.663297 31377 heartbeater.cc:511] Master 127.12.2.126:36613 requested a full tablet report, sending...
I20260620 23:09:49.664716 31060 ts_manager.cc:194] Registered new tserver with Master: b4a4efbfe9c34901a3f214df789f998b (127.12.2.66:44773)
I20260620 23:09:49.665454 31060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.66:42403
W20260620 23:09:49.966998 31380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:49.967288 31380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:49.967399 31380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:49.974570 31380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:49.974671 31380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:49.974813 31380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:09:49.983786 31380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:36613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:49.985021 31380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:49.986702 31380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:49.998960 31385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:49.998986 31388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:49.998831 31386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:49.999830 31380 server_base.cc:1034] running on GCE node
I20260620 23:09:50.000376 31380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:50.001281 31380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:50.002506 31380 hybrid_clock.cc:648] HybridClock initialized: now 1781996990002447 us; error 50 us; skew 500 ppm
I20260620 23:09:50.002825 31380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:50.005023 31380 webserver.cc:460] Webserver started at http://127.12.2.67:45873/ using document root <none> and password file <none>
I20260620 23:09:50.005716 31380 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:50.005807 31380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:50.006107 31380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:50.008917 31380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "fe5dde5848004a38a9cdcf832d781c92"
format_stamp: "Formatted at 2026-06-20 23:09:50 on dist-test-slave-7g8g"
server_key: "188ace97adac5d89c124fb31eb9a8bab"
server_key_iv: "dc1b0fa86d8b33e8da2c4be84bdd6984"
server_key_version: "encryptionkey@0"
I20260620 23:09:50.009653 31380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "fe5dde5848004a38a9cdcf832d781c92"
format_stamp: "Formatted at 2026-06-20 23:09:50 on dist-test-slave-7g8g"
server_key: "188ace97adac5d89c124fb31eb9a8bab"
server_key_iv: "dc1b0fa86d8b33e8da2c4be84bdd6984"
server_key_version: "encryptionkey@0"
I20260620 23:09:50.014474 31380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:09:50.018139 31394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:50.019773 31380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:09:50.019912 31380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "fe5dde5848004a38a9cdcf832d781c92"
format_stamp: "Formatted at 2026-06-20 23:09:50 on dist-test-slave-7g8g"
server_key: "188ace97adac5d89c124fb31eb9a8bab"
server_key_iv: "dc1b0fa86d8b33e8da2c4be84bdd6984"
server_key_version: "encryptionkey@0"
I20260620 23:09:50.020099 31380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:50.044039 31380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:50.044817 31380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:50.045069 31380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:50.045974 31380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:50.047341 31380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:50.047441 31380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:50.047542 31380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:50.047611 31380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:50.076440 31380 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:35421
I20260620 23:09:50.076493 31506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:35421 every 8 connection(s)
I20260620 23:09:50.077687 31380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:09:50.086122 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 31380
I20260620 23:09:50.086261 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:09:50.086695 12297 external_mini_cluster.cc:1428] Setting key 32a0e4bd878677a3eb0ed11bc1b0a181
I20260620 23:09:50.089576 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.080067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:49025 (local address 127.12.2.126:36613)
0620 23:09:50.080242 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:50.080246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:50.081571 (+ 1325us) server_negotiation.cc:407] Connection header received
0620 23:09:50.083265 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:50.083269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:50.083335 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:50.083416 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:50.084393 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.085111 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.086629 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.087221 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.087345 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.088693 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:50.088700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:50.088705 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:50.088726 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:50.088765 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:50.088781 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:50.088864 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:50.089130 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:09:50.089232 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:09:50.091017 31509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.080550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:36613 (local address 127.12.2.67:49025)
0620 23:09:50.081431 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:50.081466 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:50.083043 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:50.083585 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:50.083593 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:50.083685 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:50.084268 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.084284 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.085237 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.085240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.086503 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.086515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.087450 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.087456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.087649 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.088312 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:09:50.088315 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:50.088384 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:09:50.088470 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:50.088910 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:50.088919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:50.088927 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:09:50.089760 (+ 833us) client_negotiation.cc:240] Negotiation successful
0620 23:09:50.090086 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":282,"threads_started":1}
I20260620 23:09:50.092145 31507 heartbeater.cc:347] Connected to a master server at 127.12.2.126:36613
I20260620 23:09:50.092453 31507 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:50.093146 31507 heartbeater.cc:511] Master 127.12.2.126:36613 requested a full tablet report, sending...
I20260620 23:09:50.094472 31060 ts_manager.cc:194] Registered new tserver with Master: fe5dde5848004a38a9cdcf832d781c92 (127.12.2.67:35421)
I20260620 23:09:50.095238 31060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.67:49025
I20260620 23:09:50.101593 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:09:50.110730 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.105410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.12.2.126:36613)
0620 23:09:50.105584 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:50.105588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:50.106020 (+ 432us) server_negotiation.cc:407] Connection header received
0620 23:09:50.106140 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:50.106143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:50.106211 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:50.106286 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:50.106683 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.107417 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.108786 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.109174 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.109255 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.110186 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:50.110191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:50.110194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:50.110210 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:50.110239 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:50.110250 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:50.110349 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:50.110474 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 23:09:50.110514 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:09:50.114298 31060 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:09:50.120836 31060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:09:50.155548 31519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.132369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:44773 (local address 127.0.0.1:52384)
0620 23:09:50.133212 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:50.133262 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:50.133438 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:50.147342 (+ 13904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:50.147350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:50.147386 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:50.147852 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.147866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.150179 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.150188 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.152722 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.152737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.154580 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.154586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.154772 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.154827 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 23:09:50.154989 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 23:09:50.155185 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":555,"threads_started":1}
I20260620 23:09:50.157655 31520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.132809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.12.2.66:44773)
0620 23:09:50.145410 (+ 12601us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:50.145421 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:50.145460 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:09:50.145587 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:50.145597 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:50.145911 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:50.148103 (+ 2192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:50.148372 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.149781 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.152962 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.154383 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.156758 (+ 2375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.157068 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:50.157139 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 23:09:50.157317 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12420,"thread_start_us":12294,"threads_started":1}
I20260620 23:09:50.159663 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.132891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:43279 (local address 127.0.0.1:43784)
0620 23:09:50.147445 (+ 14554us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:50.147468 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:50.147579 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:50.149720 (+ 2141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:50.149724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:50.149748 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:50.150148 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.150162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.152170 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.152175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.154187 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.154203 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.158998 (+ 4795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.159004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.159226 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.159252 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:09:50.159304 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 23:09:50.159409 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14394,"mutex_wait_us":20,"thread_start_us":24372,"threads_started":1}
I20260620 23:09:50.160707 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.133150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.12.2.65:43279)
0620 23:09:50.148928 (+ 15778us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:50.148936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:50.148965 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:09:50.149063 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:50.149071 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:50.149316 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:50.149507 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:50.150349 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.152003 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.154444 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.155826 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.159898 (+ 4072us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.160142 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:50.160264 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 23:09:50.160449 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15613,"thread_start_us":16755,"threads_started":1}
I20260620 23:09:50.161594 31312 tablet_service.cc:1467] Processing CreateTablet for tablet fa017737101c46b292a2d941f6ad2eb3 (DEFAULT_TABLE table=test-table [id=dd273e586fd0452685a072c2dd8a96ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:50.162731 31519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.133779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:35421 (local address 127.0.0.1:40032)
0620 23:09:50.155811 (+ 22032us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:50.155832 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:50.155945 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:50.157003 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:50.157009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:50.157039 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:50.157199 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.157213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.158896 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.158901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.160865 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.160889 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.162052 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.162057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.162232 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.162264 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:09:50.162356 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:09:50.162483 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21917,"thread_start_us":13886,"threads_started":1}
I20260620 23:09:50.162774 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.134107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.12.2.67:35421)
0620 23:09:50.156367 (+ 22260us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:50.156376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:50.156421 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:09:50.156524 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:50.156532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:50.156808 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:50.156996 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:50.157600 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.158743 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.161032 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.161899 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.161998 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.162180 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:50.162411 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 23:09:50.162521 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22096,"thread_start_us":21949,"threads_started":1}
I20260620 23:09:50.163000 31312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa017737101c46b292a2d941f6ad2eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:50.166069 31442 tablet_service.cc:1467] Processing CreateTablet for tablet fa017737101c46b292a2d941f6ad2eb3 (DEFAULT_TABLE table=test-table [id=dd273e586fd0452685a072c2dd8a96ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:50.167418 31442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa017737101c46b292a2d941f6ad2eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:50.170205 31181 tablet_service.cc:1467] Processing CreateTablet for tablet fa017737101c46b292a2d941f6ad2eb3 (DEFAULT_TABLE table=test-table [id=dd273e586fd0452685a072c2dd8a96ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:50.171559 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa017737101c46b292a2d941f6ad2eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:50.173920 31525 tablet_bootstrap.cc:492] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b: Bootstrap starting.
I20260620 23:09:50.176764 31525 tablet_bootstrap.cc:654] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:50.178200 31525 log.cc:824] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:50.181243 31527 tablet_bootstrap.cc:492] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92: Bootstrap starting.
I20260620 23:09:50.184384 31525 tablet_bootstrap.cc:492] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b: No bootstrap required, opened a new log
I20260620 23:09:50.184662 31525 ts_tablet_manager.cc:1397] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 23:09:50.184707 31527 tablet_bootstrap.cc:654] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:50.186218 31527 log.cc:824] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:50.189198 31527 tablet_bootstrap.cc:492] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92: No bootstrap required, opened a new log
I20260620 23:09:50.189383 31528 tablet_bootstrap.cc:492] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: Bootstrap starting.
I20260620 23:09:50.189472 31527 ts_tablet_manager.cc:1397] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260620 23:09:50.192315 31528 tablet_bootstrap.cc:654] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:50.193521 31528 log.cc:824] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:50.194591 31525 raft_consensus.cc:348] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a4efbfe9c34901a3f214df789f998b" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44773 } } peers { permanent_uuid: "fe5dde5848004a38a9cdcf832d781c92" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 35421 } } peers { permanent_uuid: "6bdcfac406ce4d71b57b78497eb55a22" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43279 } }
I20260620 23:09:50.195088 31525 raft_consensus.cc:374] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:50.195194 31525 raft_consensus.cc:729] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a4efbfe9c34901a3f214df789f998b, State: Initialized, Role: FOLLOWER
I20260620 23:09:50.195905 31525 consensus_queue.cc:260] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a4efbfe9c34901a3f214df789f998b" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44773 } } peers { permanent_uuid: "fe5dde5848004a38a9cdcf832d781c92" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 35421 } } peers { permanent_uuid: "6bdcfac406ce4d71b57b78497eb55a22" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43279 } }
I20260620 23:09:50.197021 31377 heartbeater.cc:503] Master 127.12.2.126:36613 was elected leader, sending a full tablet report...
I20260620 23:09:50.197590 31525 ts_tablet_manager.cc:1428] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260620 23:09:50.199572 31527 raft_consensus.cc:348] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a4efbfe9c34901a3f214df789f998b" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44773 } } peers { permanent_uuid: "fe5dde5848004a38a9cdcf832d781c92" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 35421 } } peers { permanent_uuid: "6bdcfac406ce4d71b57b78497eb55a22" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43279 } }
I20260620 23:09:50.199887 31527 raft_consensus.cc:374] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:50.199976 31527 raft_consensus.cc:729] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe5dde5848004a38a9cdcf832d781c92, State: Initialized, Role: FOLLOWER
I20260620 23:09:50.200408 31528 tablet_bootstrap.cc:492] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: No bootstrap required, opened a new log
I20260620 23:09:50.200675 31528 ts_tablet_manager.cc:1397] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260620 23:09:50.200510 31527 consensus_queue.cc:260] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a4efbfe9c34901a3f214df789f998b" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44773 } } peers { permanent_uuid: "fe5dde5848004a38a9cdcf832d781c92" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 35421 } } peers { permanent_uuid: "6bdcfac406ce4d71b57b78497eb55a22" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43279 } }
I20260620 23:09:50.201654 31507 heartbeater.cc:503] Master 127.12.2.126:36613 was elected leader, sending a full tablet report...
I20260620 23:09:50.202131 31527 ts_tablet_manager.cc:1428] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260620 23:09:50.209249 31528 raft_consensus.cc:348] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a4efbfe9c34901a3f214df789f998b" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44773 } } peers { permanent_uuid: "fe5dde5848004a38a9cdcf832d781c92" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 35421 } } peers { permanent_uuid: "6bdcfac406ce4d71b57b78497eb55a22" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43279 } }
I20260620 23:09:50.209589 31528 raft_consensus.cc:374] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:50.209682 31528 raft_consensus.cc:729] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdcfac406ce4d71b57b78497eb55a22, State: Initialized, Role: FOLLOWER
I20260620 23:09:50.210220 31528 consensus_queue.cc:260] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a4efbfe9c34901a3f214df789f998b" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44773 } } peers { permanent_uuid: "fe5dde5848004a38a9cdcf832d781c92" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 35421 } } peers { permanent_uuid: "6bdcfac406ce4d71b57b78497eb55a22" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43279 } }
I20260620 23:09:50.211138 31246 heartbeater.cc:503] Master 127.12.2.126:36613 was elected leader, sending a full tablet report...
I20260620 23:09:50.211738 31528 ts_tablet_manager.cc:1428] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260620 23:09:50.222188 31531 raft_consensus.cc:482] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:09:50.222466 31531 raft_consensus.cc:504] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a4efbfe9c34901a3f214df789f998b" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44773 } } peers { permanent_uuid: "fe5dde5848004a38a9cdcf832d781c92" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 35421 } } peers { permanent_uuid: "6bdcfac406ce4d71b57b78497eb55a22" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43279 } }
I20260620 23:09:50.224077 31531 leader_election.cc:288] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe5dde5848004a38a9cdcf832d781c92 (127.12.2.67:35421), 6bdcfac406ce4d71b57b78497eb55a22 (127.12.2.65:43279)
W20260620 23:09:50.228369 31247 tablet.cc:2369] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:09:50.232000 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.224420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:58899 (local address 127.12.2.65:43279)
0620 23:09:50.224616 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:50.224622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:50.225512 (+ 890us) server_negotiation.cc:407] Connection header received
0620 23:09:50.225811 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:50.225816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:50.225902 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:50.225956 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:50.226736 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.227789 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.230158 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.231101 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.231211 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.231317 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:50.231577 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 23:09:50.231697 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:09:50.232043 31535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.224301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:43279 (local address 127.12.2.66:58899)
0620 23:09:50.225398 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:50.225422 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:50.225631 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:50.226132 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:50.226136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:50.226173 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:50.226590 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.226601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.227914 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.227920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.229966 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.229978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.231270 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.231277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.231441 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.231464 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:09:50.231539 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:09:50.231699 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":68,"thread_start_us":1079,"threads_started":1}
I20260620 23:09:50.233084 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.224515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:59107 (local address 127.12.2.67:35421)
0620 23:09:50.224862 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:50.224867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:50.225737 (+ 870us) server_negotiation.cc:407] Connection header received
0620 23:09:50.225880 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:50.225885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:50.225945 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:50.226015 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:50.226604 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.227681 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.230433 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.231710 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.231821 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.231911 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:50.232740 (+ 829us) server_negotiation.cc:299] Negotiation successful
0620 23:09:50.232867 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260620 23:09:50.232877 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa017737101c46b292a2d941f6ad2eb3" candidate_uuid: "b4a4efbfe9c34901a3f214df789f998b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bdcfac406ce4d71b57b78497eb55a22" is_pre_election: true
I20260620 23:09:50.233336 31534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.224379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:35421 (local address 127.12.2.66:59107)
0620 23:09:50.225639 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:50.225653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:50.225769 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:50.226040 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:50.226044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:50.226073 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:50.226454 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.226463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.227814 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.227819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.230229 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:50.230243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.232349 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.232357 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:50.232540 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.232574 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:09:50.232663 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:09:50.232781 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"mutex_wait_us":65,"thread_start_us":656,"threads_started":1}
I20260620 23:09:50.233455 31201 raft_consensus.cc:2454] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a4efbfe9c34901a3f214df789f998b in term 0.
I20260620 23:09:50.234138 31267 leader_election.cc:302] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bdcfac406ce4d71b57b78497eb55a22, b4a4efbfe9c34901a3f214df789f998b; no voters:
I20260620 23:09:50.234486 31531 raft_consensus.cc:2781] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:09:50.234340 31462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa017737101c46b292a2d941f6ad2eb3" candidate_uuid: "b4a4efbfe9c34901a3f214df789f998b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe5dde5848004a38a9cdcf832d781c92" is_pre_election: true
I20260620 23:09:50.234620 31531 raft_consensus.cc:482] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:09:50.234714 31531 raft_consensus.cc:3037] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:50.234839 31462 raft_consensus.cc:2454] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a4efbfe9c34901a3f214df789f998b in term 0.
I20260620 23:09:50.237093 31531 raft_consensus.cc:504] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a4efbfe9c34901a3f214df789f998b" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44773 } } peers { permanent_uuid: "fe5dde5848004a38a9cdcf832d781c92" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 35421 } } peers { permanent_uuid: "6bdcfac406ce4d71b57b78497eb55a22" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43279 } }
I20260620 23:09:50.237604 31531 leader_election.cc:288] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [CANDIDATE]: Term 1 election: Requested vote from peers fe5dde5848004a38a9cdcf832d781c92 (127.12.2.67:35421), 6bdcfac406ce4d71b57b78497eb55a22 (127.12.2.65:43279)
I20260620 23:09:50.237998 31462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa017737101c46b292a2d941f6ad2eb3" candidate_uuid: "b4a4efbfe9c34901a3f214df789f998b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe5dde5848004a38a9cdcf832d781c92"
I20260620 23:09:50.238093 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa017737101c46b292a2d941f6ad2eb3" candidate_uuid: "b4a4efbfe9c34901a3f214df789f998b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bdcfac406ce4d71b57b78497eb55a22"
I20260620 23:09:50.238193 31462 raft_consensus.cc:3037] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:50.238284 31201 raft_consensus.cc:3037] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:50.240530 31201 raft_consensus.cc:2454] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a4efbfe9c34901a3f214df789f998b in term 1.
I20260620 23:09:50.240770 31462 raft_consensus.cc:2454] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a4efbfe9c34901a3f214df789f998b in term 1.
I20260620 23:09:50.240901 31267 leader_election.cc:302] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bdcfac406ce4d71b57b78497eb55a22, b4a4efbfe9c34901a3f214df789f998b; no voters:
I20260620 23:09:50.241181 31531 raft_consensus.cc:2781] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:50.241559 31531 raft_consensus.cc:686] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 1 LEADER]: Becoming Leader. State: Replica: b4a4efbfe9c34901a3f214df789f998b, State: Running, Role: LEADER
I20260620 23:09:50.242075 31531 consensus_queue.cc:237] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a4efbfe9c34901a3f214df789f998b" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44773 } } peers { permanent_uuid: "fe5dde5848004a38a9cdcf832d781c92" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 35421 } } peers { permanent_uuid: "6bdcfac406ce4d71b57b78497eb55a22" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43279 } }
I20260620 23:09:50.246625 31059 catalog_manager.cc:5495] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b reported cstate change: term changed from 0 to 1, leader changed from <none> to b4a4efbfe9c34901a3f214df789f998b (127.12.2.66). New cstate: current_term: 1 leader_uuid: "b4a4efbfe9c34901a3f214df789f998b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a4efbfe9c34901a3f214df789f998b" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 44773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe5dde5848004a38a9cdcf832d781c92" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 35421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bdcfac406ce4d71b57b78497eb55a22" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43279 } health_report { overall_health: UNKNOWN } } }
I20260620 23:09:50.266245 31520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.261085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.12.2.66:44773)
0620 23:09:50.261257 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:50.261262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:50.261342 (+ 80us) server_negotiation.cc:407] Connection header received
0620 23:09:50.261545 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:50.261549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:50.261630 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:50.261749 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:09:50.262129 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.262818 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.264181 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.264576 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.264681 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.265230 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:09:50.265701 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:09:50.265846 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:50.266009 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:09:50.266051 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:09:50.274164 31462 raft_consensus.cc:1261] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92 [term 1 FOLLOWER]: Refusing update from remote peer b4a4efbfe9c34901a3f214df789f998b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:50.274164 31201 raft_consensus.cc:1261] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22 [term 1 FOLLOWER]: Refusing update from remote peer b4a4efbfe9c34901a3f214df789f998b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:50.275235 31536 consensus_queue.cc:1035] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bdcfac406ce4d71b57b78497eb55a22" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:09:50.275542 31531 consensus_queue.cc:1035] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe5dde5848004a38a9cdcf832d781c92" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 35421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:09:50.285256 31539 mvcc.cc:205] Tried to move back new op lower bound from 7299059672152461312 to 7299059672035459072. Current Snapshot: MvccSnapshot[applied={T|T < 7299059672152461312}]
I20260620 23:09:50.290730 31540 mvcc.cc:205] Tried to move back new op lower bound from 7299059672152461312 to 7299059672035459072. Current Snapshot: MvccSnapshot[applied={T|T < 7299059672152461312}]
I20260620 23:09:50.300118 31060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:50.300402 31060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:50.304056 31060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a02c07818e74029a83a2d427b2efb9a: Sending DeleteTablet for 3 replicas of tablet fa017737101c46b292a2d941f6ad2eb3
I20260620 23:09:50.305099 31312 tablet_service.cc:1514] Processing DeleteTablet for tablet fa017737101c46b292a2d941f6ad2eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:50 UTC) from {username='slave'} at 127.0.0.1:52384
I20260620 23:09:50.305243 31442 tablet_service.cc:1514] Processing DeleteTablet for tablet fa017737101c46b292a2d941f6ad2eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:50 UTC) from {username='slave'} at 127.0.0.1:40032
I20260620 23:09:50.305243 31181 tablet_service.cc:1514] Processing DeleteTablet for tablet fa017737101c46b292a2d941f6ad2eb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:50 UTC) from {username='slave'} at 127.0.0.1:43784
I20260620 23:09:50.305848 31548 tablet_replica.cc:330] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b: stopping tablet replica
I20260620 23:09:50.306173 31548 raft_consensus.cc:2229] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:09:50.306650 31548 raft_consensus.cc:2258] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:50.308233 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 31119
I20260620 23:09:50.308597 31548 ts_tablet_manager.cc:1905] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:50.315683 31548 ts_tablet_manager.cc:1918] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:50.315843 31548 log.cc:1196] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/fa017737101c46b292a2d941f6ad2eb3
I20260620 23:09:50.316349 31548 ts_tablet_manager.cc:1939] T fa017737101c46b292a2d941f6ad2eb3 P b4a4efbfe9c34901a3f214df789f998b: Deleting consensus metadata
I20260620 23:09:50.317765 31550 tablet_replica.cc:330] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: stopping tablet replica
I20260620 23:09:50.318225 31550 raft_consensus.cc:2229] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:50.318230 31047 catalog_manager.cc:4841] TS b4a4efbfe9c34901a3f214df789f998b (127.12.2.66:44773): tablet fa017737101c46b292a2d941f6ad2eb3 (table test-table [id=dd273e586fd0452685a072c2dd8a96ce]) successfully deleted
I20260620 23:09:50.318732 31550 raft_consensus.cc:2258] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:50.319558 31549 tablet_replica.cc:330] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92: stopping tablet replica
I20260620 23:09:50.320032 31549 raft_consensus.cc:2229] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:50.320549 31549 raft_consensus.cc:2258] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:50.320827 31550 ts_tablet_manager.cc:1905] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:50.322329 31549 ts_tablet_manager.cc:1905] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:50.328047 31550 ts_tablet_manager.cc:1918] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:50.328199 31550 log.cc:1196] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/fa017737101c46b292a2d941f6ad2eb3
I20260620 23:09:50.328713 31550 ts_tablet_manager.cc:1939] T fa017737101c46b292a2d941f6ad2eb3 P 6bdcfac406ce4d71b57b78497eb55a22: Deleting consensus metadata
I20260620 23:09:50.328948 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.309533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43796 (local address 127.12.2.65:43279)
0620 23:09:50.309696 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:50.309701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:50.320652 (+ 10951us) server_negotiation.cc:407] Connection header received
0620 23:09:50.321106 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:50.321112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:50.321211 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:50.321314 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:50.321925 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.323069 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.325096 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.325739 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.325839 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.327177 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:50.327187 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:50.327204 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:50.327928 (+ 724us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:50.327984 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:50.328006 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:50.328091 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:50.328432 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 23:09:50.328550 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:09:50.328878 31549 ts_tablet_manager.cc:1918] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:50.329193 31549 log.cc:1196] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/fa017737101c46b292a2d941f6ad2eb3
I20260620 23:09:50.329865 31549 ts_tablet_manager.cc:1939] T fa017737101c46b292a2d941f6ad2eb3 P fe5dde5848004a38a9cdcf832d781c92: Deleting consensus metadata
I20260620 23:09:50.330309 31241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:50.330648 31046 catalog_manager.cc:4841] TS 6bdcfac406ce4d71b57b78497eb55a22 (127.12.2.65:43279): tablet fa017737101c46b292a2d941f6ad2eb3 (table test-table [id=dd273e586fd0452685a072c2dd8a96ce]) successfully deleted
I20260620 23:09:50.331768 31047 catalog_manager.cc:4841] TS fe5dde5848004a38a9cdcf832d781c92 (127.12.2.67:35421): tablet fa017737101c46b292a2d941f6ad2eb3 (table test-table [id=dd273e586fd0452685a072c2dd8a96ce]) successfully deleted
I20260620 23:09:50.417395 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 31119
I20260620 23:09:50.431547 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 31250
I20260620 23:09:50.439193 31520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.432264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.12.2.66:44773)
0620 23:09:50.432449 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:50.432455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:50.432517 (+ 62us) server_negotiation.cc:407] Connection header received
0620 23:09:50.432702 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:50.432708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:50.432845 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:50.432930 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:50.433350 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.434522 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.435767 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.436406 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.436535 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.437453 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:50.437466 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:50.437485 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:50.438296 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:50.438363 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:50.438387 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:50.438489 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:50.438583 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:09:50.438680 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:09:50.439839 31372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:50.547165 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 31250
I20260620 23:09:50.562114 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 31380
I20260620 23:09:50.568239 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:50.562798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.12.2.67:35421)
0620 23:09:50.562978 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:50.562982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:50.562999 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:09:50.563150 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:50.563154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:50.563233 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:50.563315 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:50.563631 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.564347 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.565576 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:50.566029 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:50.566127 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:50.567061 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:50.567069 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:50.567084 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:50.567660 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:50.567706 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:50.567726 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:50.567806 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:50.567898 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 23:09:50.567981 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:09:50.568908 31502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:50.655180 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 31380
I20260620 23:09:50.667975 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 31029
I20260620 23:09:50.669001 31090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:50.774372 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 31029
2026-06-20T23:09:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2550 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:09:50 dist-test-slave-7g8g krb5kdc[31558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:09:50 dist-test-slave-7g8g krb5kdc[31558](info): set up 2 sockets
Jun 20 23:09:50 dist-test-slave-7g8g krb5kdc[31558](info): commencing operation
krb5kdc: starting...
W20260620 23:09:51.786921 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:09:51 dist-test-slave-7g8g krb5kdc[31558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:09:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:09:52.015923 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:42859
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:42859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:52.333528 31571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:52.333818 31571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:52.333959 31571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:52.341147 31571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:09:52.341254 31571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:52.341311 31571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:52.341389 31571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:09:52.341440 31571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:09:52.351275 31571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:42859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:42859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:52.352596 31571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:52.353935 31571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:52.365654 31576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:52.365695 31579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:52.365733 31577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:52.366343 31571 server_base.cc:1034] running on GCE node
I20260620 23:09:52.367190 31571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:52.368642 31571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:52.369853 31571 hybrid_clock.cc:648] HybridClock initialized: now 1781996992369793 us; error 52 us; skew 500 ppm
I20260620 23:09:52.370208 31571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:52 dist-test-slave-7g8g krb5kdc[31558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:52.373179 31571 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:09:52.374698 31571 webserver.cc:460] Webserver started at http://127.12.2.126:39745/ using document root <none> and password file <none>
I20260620 23:09:52.375413 31571 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:52.375530 31571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:52.375809 31571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:52.378744 31571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "30f0f8a0d96b4ea4b0e39a2a1be35f6b"
format_stamp: "Formatted at 2026-06-20 23:09:52 on dist-test-slave-7g8g"
server_key: "a9cd5db67a1248d65e4ced6e986827e4"
server_key_iv: "928ad21ed03a559744d342d70abee5fe"
server_key_version: "encryptionkey@0"
I20260620 23:09:52.379508 31571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "30f0f8a0d96b4ea4b0e39a2a1be35f6b"
format_stamp: "Formatted at 2026-06-20 23:09:52 on dist-test-slave-7g8g"
server_key: "a9cd5db67a1248d65e4ced6e986827e4"
server_key_iv: "928ad21ed03a559744d342d70abee5fe"
server_key_version: "encryptionkey@0"
I20260620 23:09:52.384629 31571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:09:52.388074 31586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:52.389369 31571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:09:52.389503 31571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "30f0f8a0d96b4ea4b0e39a2a1be35f6b"
format_stamp: "Formatted at 2026-06-20 23:09:52 on dist-test-slave-7g8g"
server_key: "a9cd5db67a1248d65e4ced6e986827e4"
server_key_iv: "928ad21ed03a559744d342d70abee5fe"
server_key_version: "encryptionkey@0"
I20260620 23:09:52.389662 31571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:52.414773 31571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:52.418565 31571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:52.418785 31571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:52.435636 31571 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:42859
I20260620 23:09:52.435657 31637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:42859 every 8 connection(s)
I20260620 23:09:52.436959 31571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:09:52.438671 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 31571
I20260620 23:09:52.438858 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:09:52.439426 12297 external_mini_cluster.cc:1428] Setting key 83e7779c503862fc7466c744b2420dce
I20260620 23:09:52.441270 31638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:09:52 dist-test-slave-7g8g krb5kdc[31558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:52.453979 31638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b: Bootstrap starting.
I20260620 23:09:52.457230 31638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:52.458181 31638 log.cc:824] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:52.461090 31638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b: No bootstrap required, opened a new log
I20260620 23:09:52.461185 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:52.441215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.12.2.126:42859)
0620 23:09:52.441921 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:52.441932 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:52.441972 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:09:52.443597 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:52.443618 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:52.443986 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:52.444322 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:52.446109 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:52.447447 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:52.449122 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:52.449724 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:52.449846 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:52.454434 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:52.454441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:52.454454 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:52.454493 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:52.457266 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:52.457918 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:52.457924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:52.457930 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:52.458105 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:52.458492 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:52.458497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:52.458499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:52.459222 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:52.459467 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:52.459914 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 23:09:52.460249 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":327,"threads_started":1}
I20260620 23:09:52.468195 31638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0f8a0d96b4ea4b0e39a2a1be35f6b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42859 } }
I20260620 23:09:52.468555 31638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:52.468650 31638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f0f8a0d96b4ea4b0e39a2a1be35f6b, State: Initialized, Role: FOLLOWER
I20260620 23:09:52.469194 31638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0f8a0d96b4ea4b0e39a2a1be35f6b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42859 } }
I20260620 23:09:52.469422 31638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:09:52.469544 31638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:09:52.469728 31638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:52.471870 31638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0f8a0d96b4ea4b0e39a2a1be35f6b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42859 } }
I20260620 23:09:52.472288 31638 leader_election.cc:302] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f0f8a0d96b4ea4b0e39a2a1be35f6b; no voters:
I20260620 23:09:52.472759 31638 leader_election.cc:288] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:09:52.472879 31643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:52.473448 31643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [term 1 LEADER]: Becoming Leader. State: Replica: 30f0f8a0d96b4ea4b0e39a2a1be35f6b, State: Running, Role: LEADER
I20260620 23:09:52.473937 31643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0f8a0d96b4ea4b0e39a2a1be35f6b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42859 } }
I20260620 23:09:52.474599 31638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:09:52.476605 31645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f0f8a0d96b4ea4b0e39a2a1be35f6b. Latest consensus state: current_term: 1 leader_uuid: "30f0f8a0d96b4ea4b0e39a2a1be35f6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0f8a0d96b4ea4b0e39a2a1be35f6b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42859 } } }
I20260620 23:09:52.477041 31645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:52.476630 31644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f0f8a0d96b4ea4b0e39a2a1be35f6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0f8a0d96b4ea4b0e39a2a1be35f6b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 42859 } } }
I20260620 23:09:52.477510 31644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:52.479472 31650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:09:52.483563 31650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:09:52.491675 31650 catalog_manager.cc:1347] Generated new cluster ID: ba4cc38fb098460698b945dea0ebe332
I20260620 23:09:52.491775 31650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:09:52.528301 31650 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:09:52.530154 31650 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:09:52.538015 31650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b: Generated new TSK 0
I20260620 23:09:52.538893 31650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.2.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
Entry for principal kudu/127.12.2.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab.
I20260620 23:09:52.604264 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42859
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:52.918200 31664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:52.918478 31664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:52.918604 31664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:52.925961 31664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:52.926075 31664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:52.926224 31664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:09:52.935997 31664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.65.keytab
--principal=kudu/127.12.2.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:52.937249 31664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:52.938504 31664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:52.950902 31672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:52.950893 31670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:52.950893 31669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:52.951687 31664 server_base.cc:1034] running on GCE node
I20260620 23:09:52.952247 31664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:52.953096 31664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:52.954278 31664 hybrid_clock.cc:648] HybridClock initialized: now 1781996992954220 us; error 52 us; skew 500 ppm
I20260620 23:09:52.954571 31664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:52 dist-test-slave-7g8g krb5kdc[31558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:52.960510 31664 init.cc:377] Logged in from keytab as kudu/127.12.2.65@KRBTEST.COM (short username kudu)
I20260620 23:09:52.962121 31664 webserver.cc:460] Webserver started at http://127.12.2.65:33605/ using document root <none> and password file <none>
I20260620 23:09:52.962886 31664 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:52.963032 31664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:52.963320 31664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:52.966244 31664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "11324d2588264e2794780ce364e16e20"
format_stamp: "Formatted at 2026-06-20 23:09:52 on dist-test-slave-7g8g"
server_key: "4acc62d2eb4a611ced847bfd34d24418"
server_key_iv: "d884d2ca0dd309fd9932631c1112237f"
server_key_version: "encryptionkey@0"
I20260620 23:09:52.966988 31664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "11324d2588264e2794780ce364e16e20"
format_stamp: "Formatted at 2026-06-20 23:09:52 on dist-test-slave-7g8g"
server_key: "4acc62d2eb4a611ced847bfd34d24418"
server_key_iv: "d884d2ca0dd309fd9932631c1112237f"
server_key_version: "encryptionkey@0"
I20260620 23:09:52.972047 31664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 23:09:52.975453 31679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:52.976774 31664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:09:52.976904 31664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "11324d2588264e2794780ce364e16e20"
format_stamp: "Formatted at 2026-06-20 23:09:52 on dist-test-slave-7g8g"
server_key: "4acc62d2eb4a611ced847bfd34d24418"
server_key_iv: "d884d2ca0dd309fd9932631c1112237f"
server_key_version: "encryptionkey@0"
I20260620 23:09:52.977065 31664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:52.990738 31664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:52.994886 31664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:52.995224 31664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:52.996240 31664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:52.997694 31664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:52.997777 31664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:52.997988 31664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:52.998054 31664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:53.027925 31664 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:39837
I20260620 23:09:53.027973 31791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:39837 every 8 connection(s)
I20260620 23:09:53.029202 31664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:09:53.037184 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 31664
I20260620 23:09:53.037364 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:09:53.037770 12297 external_mini_cluster.cc:1428] Setting key 60e648f8c1604b36c7ae51d71ef86e32
Jun 20 23:09:53 dist-test-slave-7g8g krb5kdc[31558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.65@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:53.049633 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:53.033225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:35753 (local address 127.12.2.126:42859)
0620 23:09:53.033672 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:53.033677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:53.033697 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:09:53.035473 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:53.035479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:53.035562 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:53.035653 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:53.037209 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:53.038143 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:53.040012 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:53.040563 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:53.040676 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:53.044231 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:53.044236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:53.044240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:53.044271 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:53.046604 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:53.047442 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:53.047448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:53.047451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:53.047597 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:53.047947 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:53.047951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:53.047960 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:53.048283 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:53.048414 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:53.049071 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 23:09:53.049288 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":227,"threads_started":1}
I20260620 23:09:53.050585 31794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:53.032665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:42859 (local address 127.12.2.65:35753)
0620 23:09:53.033466 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:53.033516 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:53.035217 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:53.035890 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:53.035898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:53.036365 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:53.037030 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:53.037053 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:53.038295 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:53.038301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:53.039876 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:53.039888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:53.040770 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:53.040778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:53.040992 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:53.041607 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:09:53.041628 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:53.044021 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:53.046740 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:53.046748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:53.046765 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:53.047326 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:53.047681 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:53.047684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:53.047686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:53.047855 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:53.048380 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:53.048386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:53.048707 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 23:09:53.048932 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 23:09:53.049463 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":319,"threads_started":1}
I20260620 23:09:53.052132 31792 heartbeater.cc:347] Connected to a master server at 127.12.2.126:42859
I20260620 23:09:53.052438 31792 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:53.053136 31792 heartbeater.cc:511] Master 127.12.2.126:42859 requested a full tablet report, sending...
I20260620 23:09:53.055178 31603 ts_manager.cc:194] Registered new tserver with Master: 11324d2588264e2794780ce364e16e20 (127.12.2.65:39837)
I20260620 23:09:53.056634 31603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.65@KRBTEST.COM'} at 127.12.2.65:35753
WARNING: no policy specified for kudu/127.12.2.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
Entry for principal kudu/127.12.2.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab.
I20260620 23:09:53.087870 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42859
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:53.398901 31798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:53.399196 31798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:53.399333 31798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:53.406355 31798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:53.406455 31798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:53.406601 31798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:09:53.415906 31798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.66.keytab
--principal=kudu/127.12.2.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:53.417129 31798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:53.418421 31798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:53.430300 31803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:53.430325 31804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:53.430356 31806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:53.431157 31798 server_base.cc:1034] running on GCE node
I20260620 23:09:53.431715 31798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:53.432649 31798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:53.433883 31798 hybrid_clock.cc:648] HybridClock initialized: now 1781996993433828 us; error 46 us; skew 500 ppm
I20260620 23:09:53.434170 31798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:53 dist-test-slave-7g8g krb5kdc[31558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996993, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:53.437130 31798 init.cc:377] Logged in from keytab as kudu/127.12.2.66@KRBTEST.COM (short username kudu)
I20260620 23:09:53.438616 31798 webserver.cc:460] Webserver started at http://127.12.2.66:44413/ using document root <none> and password file <none>
I20260620 23:09:53.439289 31798 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:53.439400 31798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:53.439683 31798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:53.442451 31798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "7f735aa36ad24a96b849412ef8e03e03"
format_stamp: "Formatted at 2026-06-20 23:09:53 on dist-test-slave-7g8g"
server_key: "bc26b2ae7471a1c895d26511d402f6b1"
server_key_iv: "f78b4875d6404a1c19f0b220c66aa55c"
server_key_version: "encryptionkey@0"
I20260620 23:09:53.443197 31798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "7f735aa36ad24a96b849412ef8e03e03"
format_stamp: "Formatted at 2026-06-20 23:09:53 on dist-test-slave-7g8g"
server_key: "bc26b2ae7471a1c895d26511d402f6b1"
server_key_iv: "f78b4875d6404a1c19f0b220c66aa55c"
server_key_version: "encryptionkey@0"
I20260620 23:09:53.448396 31798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 23:09:53.451707 31813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:53.452955 31798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:09:53.453091 31798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "7f735aa36ad24a96b849412ef8e03e03"
format_stamp: "Formatted at 2026-06-20 23:09:53 on dist-test-slave-7g8g"
server_key: "bc26b2ae7471a1c895d26511d402f6b1"
server_key_iv: "f78b4875d6404a1c19f0b220c66aa55c"
server_key_version: "encryptionkey@0"
I20260620 23:09:53.453258 31798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:53.479563 31798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:53.483394 31798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:53.483639 31798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:53.484547 31798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:53.485759 31798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:53.485828 31798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:53.485940 31798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:53.485986 31798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:53.514267 31798 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:42993
I20260620 23:09:53.514307 31925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:42993 every 8 connection(s)
I20260620 23:09:53.515475 31798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:09:53.520031 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 31798
I20260620 23:09:53.520212 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:09:53.520725 12297 external_mini_cluster.cc:1428] Setting key 960c98845e5b8be2bff84f3bfe28dc9b
Jun 20 23:09:53 dist-test-slave-7g8g krb5kdc[31558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996993, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.66@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:53.535821 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:53.517755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:37531 (local address 127.12.2.126:42859)
0620 23:09:53.517978 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:53.517985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:53.519080 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 23:09:53.521328 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:53.521333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:53.521422 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:53.521541 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:53.523196 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:53.523916 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:53.525956 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:53.526365 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:53.526465 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:53.530142 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:53.530147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:53.530150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:53.530185 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:53.532682 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:53.533503 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:53.533506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:53.533509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:53.533637 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:53.534112 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:53.534116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:53.534119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:53.534442 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:53.534583 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:53.535391 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 23:09:53.535573 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:09:53.537068 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:53.518252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:42859 (local address 127.12.2.66:37531)
0620 23:09:53.518902 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:53.518968 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:53.521009 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:53.521780 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:53.521791 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:53.522289 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:53.523037 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:53.523062 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:53.524072 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:53.524077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:53.525806 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:53.525820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:53.526562 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:53.526568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:53.526815 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:53.527510 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:09:53.527535 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:53.529930 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:53.532824 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:53.532832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:53.532850 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:53.533379 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:53.533731 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:53.533736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:53.533740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:53.533959 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:53.534570 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:53.534578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:53.534995 (+ 417us) client_negotiation.cc:769] Sending connection context
0620 23:09:53.535266 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 23:09:53.535774 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":258,"threads_started":1}
I20260620 23:09:53.538552 31926 heartbeater.cc:347] Connected to a master server at 127.12.2.126:42859
I20260620 23:09:53.538861 31926 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:53.539688 31926 heartbeater.cc:511] Master 127.12.2.126:42859 requested a full tablet report, sending...
I20260620 23:09:53.541138 31603 ts_manager.cc:194] Registered new tserver with Master: 7f735aa36ad24a96b849412ef8e03e03 (127.12.2.66:42993)
I20260620 23:09:53.541915 31603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.66@KRBTEST.COM'} at 127.12.2.66:37531
WARNING: no policy specified for kudu/127.12.2.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
Entry for principal kudu/127.12.2.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab.
I20260620 23:09:53.575290 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42859
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:53.880748 31931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:53.881031 31931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:53.881167 31931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:53.888196 31931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:53.888309 31931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:53.888450 31931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:09:53.898010 31931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.67.keytab
--principal=kudu/127.12.2.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:53.899255 31931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:53.900575 31931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:53.912473 31937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:53.912509 31939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:53.912534 31936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:53.913210 31931 server_base.cc:1034] running on GCE node
I20260620 23:09:53.913715 31931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:53.914559 31931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:53.915752 31931 hybrid_clock.cc:648] HybridClock initialized: now 1781996993915701 us; error 43 us; skew 500 ppm
I20260620 23:09:53.916045 31931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:53 dist-test-slave-7g8g krb5kdc[31558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996993, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:53.918920 31931 init.cc:377] Logged in from keytab as kudu/127.12.2.67@KRBTEST.COM (short username kudu)
I20260620 23:09:53.920295 31931 webserver.cc:460] Webserver started at http://127.12.2.67:34013/ using document root <none> and password file <none>
I20260620 23:09:53.920928 31931 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:53.921022 31931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:53.921270 31931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:53.924082 31931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "8d19db17489e4cbda94868141b795e9f"
format_stamp: "Formatted at 2026-06-20 23:09:53 on dist-test-slave-7g8g"
server_key: "3d15fe862adc9517d09cda09f32b34b5"
server_key_iv: "583acf2ae819e3e798a086736ae81720"
server_key_version: "encryptionkey@0"
I20260620 23:09:53.924762 31931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "8d19db17489e4cbda94868141b795e9f"
format_stamp: "Formatted at 2026-06-20 23:09:53 on dist-test-slave-7g8g"
server_key: "3d15fe862adc9517d09cda09f32b34b5"
server_key_iv: "583acf2ae819e3e798a086736ae81720"
server_key_version: "encryptionkey@0"
I20260620 23:09:53.929548 31931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 23:09:53.932729 31946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:53.933960 31931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:09:53.934093 31931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "8d19db17489e4cbda94868141b795e9f"
format_stamp: "Formatted at 2026-06-20 23:09:53 on dist-test-slave-7g8g"
server_key: "3d15fe862adc9517d09cda09f32b34b5"
server_key_iv: "583acf2ae819e3e798a086736ae81720"
server_key_version: "encryptionkey@0"
I20260620 23:09:53.934267 31931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:53.955281 31931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:53.959576 31931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:53.959839 31931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:53.960737 31931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:53.962062 31931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:53.962136 31931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:53.962251 31931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:53.962303 31931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:53.990667 31931 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:40697
I20260620 23:09:53.990715 32058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:40697 every 8 connection(s)
I20260620 23:09:53.992033 31931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:09:53.996794 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 31931
I20260620 23:09:53.996989 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:09:53.997499 12297 external_mini_cluster.cc:1428] Setting key 173fd4ac00f6bf3dfab6f023d9011e9f
Jun 20 23:09:53 dist-test-slave-7g8g krb5kdc[31558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996993, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.67@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:09:54.011319 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:53.994506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:59973 (local address 127.12.2.126:42859)
0620 23:09:53.994674 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:53.994678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:53.995868 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 23:09:53.998230 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:53.998235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:53.998340 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:53.998425 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:54.000037 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.000785 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.002299 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.002696 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.002804 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.005999 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:54.006004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:54.006007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:54.006041 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:54.008379 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:54.009068 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:54.009072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:54.009075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:54.009196 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:54.009597 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:54.009601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:54.009603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:54.009878 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:54.010016 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:54.010689 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 23:09:54.010911 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:09:54.012233 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:53.995033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:42859 (local address 127.12.2.67:59973)
0620 23:09:53.995712 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:53.995748 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:53.997960 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:53.998667 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:53.998679 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:53.999282 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:53.999881 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:53.999899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.000929 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.000934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.002169 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:54.002180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.002899 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.002906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.003163 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.003724 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:09:54.003745 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:54.005801 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:54.008513 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:54.008523 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:54.008541 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:54.008948 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:54.009293 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:09:54.009298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:09:54.009302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:09:54.009491 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:09:54.009994 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:54.010001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:54.010336 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 23:09:54.010575 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 23:09:54.011132 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":253,"threads_started":1}
I20260620 23:09:54.013624 32059 heartbeater.cc:347] Connected to a master server at 127.12.2.126:42859
I20260620 23:09:54.013899 32059 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:54.014561 32059 heartbeater.cc:511] Master 127.12.2.126:42859 requested a full tablet report, sending...
I20260620 23:09:54.015957 31602 ts_manager.cc:194] Registered new tserver with Master: 8d19db17489e4cbda94868141b795e9f (127.12.2.67:40697)
I20260620 23:09:54.016808 31602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.2.67@KRBTEST.COM'} at 127.12.2.67:59973
I20260620 23:09:54.025389 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:09:54.042804 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.029626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.12.2.126:42859)
0620 23:09:54.029818 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:54.029825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:54.030169 (+ 344us) server_negotiation.cc:407] Connection header received
0620 23:09:54.030313 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:54.030318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:54.030395 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:54.030497 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:54.032042 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.033106 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.034906 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.035545 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.035681 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.038127 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:54.038132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:54.038135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:54.038169 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:54.040688 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:54.041297 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:54.041300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:54.041303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:54.041422 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:54.041700 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:54.041703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:54.041705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:54.041989 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:54.042163 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:54.042414 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 23:09:54.042575 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:09:54.046604 31602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:09:54.053431 31602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:09:54.059522 31792 heartbeater.cc:503] Master 127.12.2.126:42859 was elected leader, sending a full tablet report...
I20260620 23:09:54.076737 32072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.065159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:42993 (local address 127.0.0.1:50230)
0620 23:09:54.067033 (+ 1874us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:54.067049 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:54.067152 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:54.068138 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:54.068146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:54.068185 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:54.069218 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:54.069235 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.071232 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.071238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.073925 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:54.073941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.075627 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.075637 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.075830 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.075882 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:09:54.076021 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:09:54.076212 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1728,"mutex_wait_us":41,"spinlock_wait_cycles":896,"thread_start_us":1587,"threads_started":1}
I20260620 23:09:54.076880 32071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.065040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:40697 (local address 127.0.0.1:33634)
0620 23:09:54.066432 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:54.066473 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:54.066614 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:54.068138 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:54.068146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:54.068190 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:54.069033 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:54.069047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.071191 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.071197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.074248 (+ 3051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:54.074258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.075974 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.075981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.076157 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.076189 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:09:54.076281 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:09:54.076420 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":58,"spinlock_wait_cycles":3456,"thread_start_us":1029,"threads_started":1}
I20260620 23:09:54.077379 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.065596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.12.2.66:42993)
0620 23:09:54.067577 (+ 1981us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:54.067585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:54.067609 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:09:54.067692 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:54.067698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:54.067914 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:54.068261 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:54.069420 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.071062 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.074089 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.075457 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.075566 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.075785 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:54.076993 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0620 23:09:54.077159 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1839,"thread_start_us":1751,"threads_started":1}
I20260620 23:09:54.077862 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.065249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.12.2.67:40697)
0620 23:09:54.067257 (+ 2008us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:54.067266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:54.067300 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:09:54.067407 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:54.067414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:54.067667 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:54.067843 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:54.069252 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.071029 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.074390 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.075505 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.075622 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.075833 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:54.077411 (+ 1578us) server_negotiation.cc:299] Negotiation successful
0620 23:09:54.077570 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1836,"thread_start_us":1704,"threads_started":1}
I20260620 23:09:54.079806 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.067116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39837 (local address 127.0.0.1:56358)
0620 23:09:54.071759 (+ 4643us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:54.071777 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:54.071885 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:54.072928 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:54.072932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:54.072959 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:54.073365 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:54.073374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.075190 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.075196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.077271 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:54.077285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.079190 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.079197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.079308 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.079330 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:09:54.079399 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 23:09:54.079517 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4528,"thread_start_us":4257,"threads_started":1}
I20260620 23:09:54.079918 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.067596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.12.2.65:39837)
0620 23:09:54.072413 (+ 4817us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:54.072434 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:54.072459 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:09:54.072560 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:54.072567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:54.072755 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:54.072915 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:54.073550 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.075060 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.077423 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.078748 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.078857 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.079054 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:54.079467 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 23:09:54.079630 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4654,"thread_start_us":4514,"threads_started":1}
I20260620 23:09:54.081277 31994 tablet_service.cc:1467] Processing CreateTablet for tablet fbfef522963047bfa5c4f8c5a31ce729 (DEFAULT_TABLE table=test-table [id=851ca12a0f59424abe933b0528b783d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:54.081277 31861 tablet_service.cc:1467] Processing CreateTablet for tablet fbfef522963047bfa5c4f8c5a31ce729 (DEFAULT_TABLE table=test-table [id=851ca12a0f59424abe933b0528b783d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:54.082747 31994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfef522963047bfa5c4f8c5a31ce729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:54.082747 31861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfef522963047bfa5c4f8c5a31ce729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:54.089526 31727 tablet_service.cc:1467] Processing CreateTablet for tablet fbfef522963047bfa5c4f8c5a31ce729 (DEFAULT_TABLE table=test-table [id=851ca12a0f59424abe933b0528b783d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:54.090670 31727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfef522963047bfa5c4f8c5a31ce729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:54.097206 32077 tablet_bootstrap.cc:492] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: Bootstrap starting.
I20260620 23:09:54.097956 32078 tablet_bootstrap.cc:492] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03: Bootstrap starting.
I20260620 23:09:54.101449 32078 tablet_bootstrap.cc:654] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:54.101639 32077 tablet_bootstrap.cc:654] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:54.102540 32078 log.cc:824] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:54.103096 32079 tablet_bootstrap.cc:492] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20: Bootstrap starting.
I20260620 23:09:54.104120 32077 log.cc:824] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:54.105872 32078 tablet_bootstrap.cc:492] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03: No bootstrap required, opened a new log
I20260620 23:09:54.106067 32079 tablet_bootstrap.cc:654] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:54.106132 32078 ts_tablet_manager.cc:1397] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 23:09:54.107028 32077 tablet_bootstrap.cc:492] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: No bootstrap required, opened a new log
I20260620 23:09:54.107033 32079 log.cc:824] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:54.107301 32077 ts_tablet_manager.cc:1397] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 23:09:54.109581 32079 tablet_bootstrap.cc:492] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20: No bootstrap required, opened a new log
I20260620 23:09:54.109793 32079 ts_tablet_manager.cc:1397] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 23:09:54.116415 32078 raft_consensus.cc:348] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d19db17489e4cbda94868141b795e9f" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40697 } } peers { permanent_uuid: "7f735aa36ad24a96b849412ef8e03e03" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 42993 } } peers { permanent_uuid: "11324d2588264e2794780ce364e16e20" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39837 } }
I20260620 23:09:54.116896 32078 raft_consensus.cc:374] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:54.116999 32078 raft_consensus.cc:729] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f735aa36ad24a96b849412ef8e03e03, State: Initialized, Role: FOLLOWER
I20260620 23:09:54.116801 32079 raft_consensus.cc:348] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d19db17489e4cbda94868141b795e9f" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40697 } } peers { permanent_uuid: "7f735aa36ad24a96b849412ef8e03e03" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 42993 } } peers { permanent_uuid: "11324d2588264e2794780ce364e16e20" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39837 } }
I20260620 23:09:54.117193 32079 raft_consensus.cc:374] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:54.117280 32079 raft_consensus.cc:729] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11324d2588264e2794780ce364e16e20, State: Initialized, Role: FOLLOWER
I20260620 23:09:54.117300 32077 raft_consensus.cc:348] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d19db17489e4cbda94868141b795e9f" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40697 } } peers { permanent_uuid: "7f735aa36ad24a96b849412ef8e03e03" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 42993 } } peers { permanent_uuid: "11324d2588264e2794780ce364e16e20" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39837 } }
I20260620 23:09:54.117681 32077 raft_consensus.cc:374] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:54.117772 32077 raft_consensus.cc:729] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d19db17489e4cbda94868141b795e9f, State: Initialized, Role: FOLLOWER
I20260620 23:09:54.117654 32078 consensus_queue.cc:260] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d19db17489e4cbda94868141b795e9f" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40697 } } peers { permanent_uuid: "7f735aa36ad24a96b849412ef8e03e03" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 42993 } } peers { permanent_uuid: "11324d2588264e2794780ce364e16e20" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39837 } }
I20260620 23:09:54.117853 32079 consensus_queue.cc:260] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d19db17489e4cbda94868141b795e9f" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40697 } } peers { permanent_uuid: "7f735aa36ad24a96b849412ef8e03e03" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 42993 } } peers { permanent_uuid: "11324d2588264e2794780ce364e16e20" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39837 } }
I20260620 23:09:54.118320 32077 consensus_queue.cc:260] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d19db17489e4cbda94868141b795e9f" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40697 } } peers { permanent_uuid: "7f735aa36ad24a96b849412ef8e03e03" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 42993 } } peers { permanent_uuid: "11324d2588264e2794780ce364e16e20" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39837 } }
I20260620 23:09:54.119648 32079 ts_tablet_manager.cc:1428] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 23:09:54.122469 32059 heartbeater.cc:503] Master 127.12.2.126:42859 was elected leader, sending a full tablet report...
I20260620 23:09:54.123472 32077 ts_tablet_manager.cc:1428] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260620 23:09:54.124365 31926 heartbeater.cc:503] Master 127.12.2.126:42859 was elected leader, sending a full tablet report...
I20260620 23:09:54.125006 32078 ts_tablet_manager.cc:1428] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03: Time spent starting tablet: real 0.019s user 0.011s sys 0.008s
I20260620 23:09:54.161128 32083 raft_consensus.cc:482] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:09:54.161430 32083 raft_consensus.cc:504] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d19db17489e4cbda94868141b795e9f" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40697 } } peers { permanent_uuid: "7f735aa36ad24a96b849412ef8e03e03" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 42993 } } peers { permanent_uuid: "11324d2588264e2794780ce364e16e20" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39837 } }
I20260620 23:09:54.163159 32083 leader_election.cc:288] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d19db17489e4cbda94868141b795e9f (127.12.2.67:40697), 11324d2588264e2794780ce364e16e20 (127.12.2.65:39837)
I20260620 23:09:54.170202 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.164218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:45867 (local address 127.12.2.67:40697)
0620 23:09:54.164616 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:54.164621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:54.164936 (+ 315us) server_negotiation.cc:407] Connection header received
0620 23:09:54.165163 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:54.165169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:54.165246 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:54.165332 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:54.165983 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.166692 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.168533 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.169300 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.169423 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.169528 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:54.169830 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 23:09:54.169957 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260620 23:09:54.170208 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.163722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:58399 (local address 127.12.2.65:39837)
0620 23:09:54.164455 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:54.164462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:54.165270 (+ 808us) server_negotiation.cc:407] Connection header received
0620 23:09:54.165433 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:54.165440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:54.165528 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:54.165606 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:54.166142 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.166814 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.168538 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.169306 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.169424 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.169530 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:54.169845 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 23:09:54.169979 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570}
I20260620 23:09:54.170403 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.163501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:40697 (local address 127.12.2.66:45867)
0620 23:09:54.164821 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:54.164842 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:54.165003 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:54.165432 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:54.165437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:54.165459 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:54.165798 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:54.165812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.166813 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.166817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.168393 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:54.168404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.169531 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.169541 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.169684 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.169714 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:09:54.169790 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:09:54.169950 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192,"mutex_wait_us":25,"thread_start_us":722,"threads_started":1}
I20260620 23:09:54.170406 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.163419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:39837 (local address 127.12.2.66:58399)
0620 23:09:54.165165 (+ 1746us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:54.165186 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:54.165286 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:54.165644 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:54.165648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:54.165675 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:54.166009 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:54.166017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.166966 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.166971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.168394 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:54.168406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.169556 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.169563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:54.169704 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.169726 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:09:54.169801 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:09:54.169960 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":50,"thread_start_us":1117,"threads_started":1}
I20260620 23:09:54.171476 31747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfef522963047bfa5c4f8c5a31ce729" candidate_uuid: "7f735aa36ad24a96b849412ef8e03e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11324d2588264e2794780ce364e16e20" is_pre_election: true
I20260620 23:09:54.171818 32014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfef522963047bfa5c4f8c5a31ce729" candidate_uuid: "7f735aa36ad24a96b849412ef8e03e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d19db17489e4cbda94868141b795e9f" is_pre_election: true
I20260620 23:09:54.171965 31747 raft_consensus.cc:2454] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f735aa36ad24a96b849412ef8e03e03 in term 0.
I20260620 23:09:54.172287 32014 raft_consensus.cc:2454] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f735aa36ad24a96b849412ef8e03e03 in term 0.
I20260620 23:09:54.172588 31814 leader_election.cc:302] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11324d2588264e2794780ce364e16e20, 7f735aa36ad24a96b849412ef8e03e03; no voters:
I20260620 23:09:54.172935 32083 raft_consensus.cc:2781] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:09:54.173053 32083 raft_consensus.cc:482] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:09:54.173144 32083 raft_consensus.cc:3037] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:54.175333 32083 raft_consensus.cc:504] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d19db17489e4cbda94868141b795e9f" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40697 } } peers { permanent_uuid: "7f735aa36ad24a96b849412ef8e03e03" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 42993 } } peers { permanent_uuid: "11324d2588264e2794780ce364e16e20" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39837 } }
I20260620 23:09:54.175935 32083 leader_election.cc:288] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [CANDIDATE]: Term 1 election: Requested vote from peers 8d19db17489e4cbda94868141b795e9f (127.12.2.67:40697), 11324d2588264e2794780ce364e16e20 (127.12.2.65:39837)
I20260620 23:09:54.176343 32014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfef522963047bfa5c4f8c5a31ce729" candidate_uuid: "7f735aa36ad24a96b849412ef8e03e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d19db17489e4cbda94868141b795e9f"
I20260620 23:09:54.176411 31747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfef522963047bfa5c4f8c5a31ce729" candidate_uuid: "7f735aa36ad24a96b849412ef8e03e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11324d2588264e2794780ce364e16e20"
I20260620 23:09:54.176576 32014 raft_consensus.cc:3037] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:54.176627 31747 raft_consensus.cc:3037] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:54.179704 31747 raft_consensus.cc:2454] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f735aa36ad24a96b849412ef8e03e03 in term 1.
I20260620 23:09:54.179704 32014 raft_consensus.cc:2454] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f735aa36ad24a96b849412ef8e03e03 in term 1.
I20260620 23:09:54.180115 31817 leader_election.cc:302] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f735aa36ad24a96b849412ef8e03e03, 8d19db17489e4cbda94868141b795e9f; no voters:
I20260620 23:09:54.180389 32083 raft_consensus.cc:2781] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:54.180884 32083 raft_consensus.cc:686] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 1 LEADER]: Becoming Leader. State: Replica: 7f735aa36ad24a96b849412ef8e03e03, State: Running, Role: LEADER
I20260620 23:09:54.181406 32083 consensus_queue.cc:237] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d19db17489e4cbda94868141b795e9f" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40697 } } peers { permanent_uuid: "7f735aa36ad24a96b849412ef8e03e03" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 42993 } } peers { permanent_uuid: "11324d2588264e2794780ce364e16e20" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39837 } }
I20260620 23:09:54.185784 31602 catalog_manager.cc:5495] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f735aa36ad24a96b849412ef8e03e03 (127.12.2.66). New cstate: current_term: 1 leader_uuid: "7f735aa36ad24a96b849412ef8e03e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d19db17489e4cbda94868141b795e9f" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f735aa36ad24a96b849412ef8e03e03" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 42993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11324d2588264e2794780ce364e16e20" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39837 } health_report { overall_health: UNKNOWN } } }
I20260620 23:09:54.230332 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.225039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.12.2.66:42993)
0620 23:09:54.225291 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:54.225296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:54.225312 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:09:54.225410 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:54.225416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:54.225533 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:54.225682 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:09:54.226051 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.226808 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.228270 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.228680 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.228782 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.229266 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:09:54.229779 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:09:54.229907 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:54.229990 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 23:09:54.230071 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:09:54.238588 31747 raft_consensus.cc:1261] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20 [term 1 FOLLOWER]: Refusing update from remote peer 7f735aa36ad24a96b849412ef8e03e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:54.238590 32014 raft_consensus.cc:1261] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f [term 1 FOLLOWER]: Refusing update from remote peer 7f735aa36ad24a96b849412ef8e03e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:54.239547 32088 consensus_queue.cc:1035] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11324d2588264e2794780ce364e16e20" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:09:54.239882 32083 consensus_queue.cc:1035] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d19db17489e4cbda94868141b795e9f" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:09:54.243520 32060 tablet.cc:2369] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:09:54.253515 32093 mvcc.cc:205] Tried to move back new op lower bound from 7299059688391573504 to 7299059688170958848. Current Snapshot: MvccSnapshot[applied={T|T < 7299059688391573504}]
I20260620 23:09:54.266170 31602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:54.266412 31602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:54.270062 31602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30f0f8a0d96b4ea4b0e39a2a1be35f6b: Sending DeleteTablet for 3 replicas of tablet fbfef522963047bfa5c4f8c5a31ce729
I20260620 23:09:54.271216 31994 tablet_service.cc:1514] Processing DeleteTablet for tablet fbfef522963047bfa5c4f8c5a31ce729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:54 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:33634
I20260620 23:09:54.271225 31861 tablet_service.cc:1514] Processing DeleteTablet for tablet fbfef522963047bfa5c4f8c5a31ce729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:54 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:50230
I20260620 23:09:54.271484 31727 tablet_service.cc:1514] Processing DeleteTablet for tablet fbfef522963047bfa5c4f8c5a31ce729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:54 UTC) from {username='kudu', principal='kudu/127.12.2.126@KRBTEST.COM'} at 127.0.0.1:56358
I20260620 23:09:54.273509 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 31664
I20260620 23:09:54.287106 32102 tablet_replica.cc:330] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20: stopping tablet replica
I20260620 23:09:54.287779 32102 raft_consensus.cc:2229] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:54.288458 32102 raft_consensus.cc:2258] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:54.290768 32102 ts_tablet_manager.cc:1905] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:54.294338 32100 tablet_replica.cc:330] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: stopping tablet replica
I20260620 23:09:54.294795 32100 raft_consensus.cc:2229] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:54.295341 32100 raft_consensus.cc:2258] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:54.296147 32101 tablet_replica.cc:330] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03: stopping tablet replica
I20260620 23:09:54.296566 32101 raft_consensus.cc:2229] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:09:54.297210 32101 raft_consensus.cc:2258] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:54.299263 32101 ts_tablet_manager.cc:1905] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:54.299556 32102 ts_tablet_manager.cc:1918] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:54.299703 32102 log.cc:1196] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-0/wal/wals/fbfef522963047bfa5c4f8c5a31ce729
I20260620 23:09:54.300137 32102 ts_tablet_manager.cc:1939] T fbfef522963047bfa5c4f8c5a31ce729 P 11324d2588264e2794780ce364e16e20: Deleting consensus metadata
Jun 20 23:09:54 dist-test-slave-7g8g krb5kdc[31558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.65@KRBTEST.COM
I20260620 23:09:54.301765 31587 catalog_manager.cc:4841] TS 11324d2588264e2794780ce364e16e20 (127.12.2.65:39837): tablet fbfef522963047bfa5c4f8c5a31ce729 (table test-table [id=851ca12a0f59424abe933b0528b783d0]) successfully deleted
I20260620 23:09:54.303490 32100 ts_tablet_manager.cc:1905] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:54.306134 32101 ts_tablet_manager.cc:1918] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:54.306303 32101 log.cc:1196] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/fbfef522963047bfa5c4f8c5a31ce729
I20260620 23:09:54.306875 32101 ts_tablet_manager.cc:1939] T fbfef522963047bfa5c4f8c5a31ce729 P 7f735aa36ad24a96b849412ef8e03e03: Deleting consensus metadata
I20260620 23:09:54.308703 31587 catalog_manager.cc:4841] TS 7f735aa36ad24a96b849412ef8e03e03 (127.12.2.66:42993): tablet fbfef522963047bfa5c4f8c5a31ce729 (table test-table [id=851ca12a0f59424abe933b0528b783d0]) successfully deleted
I20260620 23:09:54.309612 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.274914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.12.2.65:39837)
0620 23:09:54.275095 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:54.275101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:54.279878 (+ 4777us) server_negotiation.cc:407] Connection header received
0620 23:09:54.280189 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:54.280196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:54.280300 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:54.280379 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:54.292969 (+ 12590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.294104 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.296090 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.296694 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.296852 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.301993 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:54.302006 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:54.302024 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:54.302066 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:54.305577 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:54.306505 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:54.306513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:54.306517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:54.306668 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:54.307130 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:54.307135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:54.307138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:54.307589 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:54.307887 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:54.309029 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0620 23:09:54.309245 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:09:54.310367 31787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:54.310597 32100 ts_tablet_manager.cc:1918] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:54.310771 32100 log.cc:1196] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/fbfef522963047bfa5c4f8c5a31ce729
I20260620 23:09:54.311331 32100 ts_tablet_manager.cc:1939] T fbfef522963047bfa5c4f8c5a31ce729 P 8d19db17489e4cbda94868141b795e9f: Deleting consensus metadata
I20260620 23:09:54.313207 31590 catalog_manager.cc:4841] TS 8d19db17489e4cbda94868141b795e9f (127.12.2.67:40697): tablet fbfef522963047bfa5c4f8c5a31ce729 (table test-table [id=851ca12a0f59424abe933b0528b783d0]) successfully deleted
I20260620 23:09:54.399550 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 31664
I20260620 23:09:54.421821 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 31798
Jun 20 23:09:54 dist-test-slave-7g8g krb5kdc[31558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.66@KRBTEST.COM
I20260620 23:09:54.440467 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.422710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.12.2.66:42993)
0620 23:09:54.422904 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:54.422910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:54.422961 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:09:54.423140 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:54.423146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:54.423235 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:54.423351 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:54.424806 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.425994 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.427501 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.428172 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.428295 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.432781 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:54.432801 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:54.432821 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:54.432862 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:54.436757 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:54.437662 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:54.437671 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:54.437675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:54.437866 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:54.438341 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:54.438347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:54.438351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:54.438885 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:54.439257 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:54.439861 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 23:09:54.440081 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:09:54.441183 31921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:54.556806 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 31798
I20260620 23:09:54.571012 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 31931
Jun 20 23:09:54 dist-test-slave-7g8g krb5kdc[31558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.67@KRBTEST.COM
I20260620 23:09:54.585181 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:54.571680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.12.2.67:40697)
0620 23:09:54.571855 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:54.571860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:54.571877 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:09:54.572028 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:54.572032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:54.572095 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:54.572175 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:54.573477 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.574193 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.575624 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:54.576084 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:54.576180 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:54.580414 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:54.580429 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:54.580445 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:54.580486 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:54.583052 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:54.583658 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:54.583665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:54.583667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:54.583782 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:54.584043 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:54.584046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:54.584047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:54.584359 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:54.584565 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:54.584756 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 23:09:54.584923 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:09:54.585796 32054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:54.672253 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 31931
I20260620 23:09:54.685209 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 31571
I20260620 23:09:54.686280 31633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:54.793623 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 31571
2026-06-20T23:09:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4020 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T23:09:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:09:54Z Disabled control of system clock
I20260620 23:09:54.866732 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43497
--webserver_interface=127.12.2.126
--webserver_port=0
--builtin_ntp_servers=127.12.2.84:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:43497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:09:55.179287 32111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:55.179582 32111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:55.179687 32111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:55.186981 32111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:09:55.187089 32111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:55.187132 32111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:55.187186 32111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:09:55.187228 32111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:09:55.196236 32111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:43497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:55.197445 32111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:55.199127 32111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:55.210038 32117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:55.210075 32119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:55.210008 32116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:55.210824 32111 server_base.cc:1034] running on GCE node
I20260620 23:09:55.211567 32111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:55.212858 32111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:55.214059 32111 hybrid_clock.cc:648] HybridClock initialized: now 1781996995214014 us; error 35 us; skew 500 ppm
I20260620 23:09:55.214381 32111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:55.216529 32111 webserver.cc:460] Webserver started at http://127.12.2.126:42185/ using document root <none> and password file <none>
I20260620 23:09:55.217161 32111 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:55.217254 32111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:55.217532 32111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:55.220288 32111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "9412bfe985ca469cbd16542d9f2c694b"
format_stamp: "Formatted at 2026-06-20 23:09:55 on dist-test-slave-7g8g"
server_key: "80d17120c3597833d9fb440d27c40430"
server_key_iv: "ddda869652a4c9d2666ce16ab564d871"
server_key_version: "encryptionkey@0"
I20260620 23:09:55.221043 32111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "9412bfe985ca469cbd16542d9f2c694b"
format_stamp: "Formatted at 2026-06-20 23:09:55 on dist-test-slave-7g8g"
server_key: "80d17120c3597833d9fb440d27c40430"
server_key_iv: "ddda869652a4c9d2666ce16ab564d871"
server_key_version: "encryptionkey@0"
I20260620 23:09:55.225859 32111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:09:55.229503 32125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:55.230818 32111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:09:55.231040 32111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "9412bfe985ca469cbd16542d9f2c694b"
format_stamp: "Formatted at 2026-06-20 23:09:55 on dist-test-slave-7g8g"
server_key: "80d17120c3597833d9fb440d27c40430"
server_key_iv: "ddda869652a4c9d2666ce16ab564d871"
server_key_version: "encryptionkey@0"
I20260620 23:09:55.231225 32111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:55.259529 32111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:55.260433 32111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:55.260703 32111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:55.277637 32111 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:43497
I20260620 23:09:55.277643 32176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:43497 every 8 connection(s)
I20260620 23:09:55.278898 32111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:09:55.280007 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 32111
I20260620 23:09:55.280182 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:09:55.280712 12297 external_mini_cluster.cc:1428] Setting key aafb5b0ae9735219f3d16e270dee2e1a
I20260620 23:09:55.284489 32177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:55.293143 32180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:55.282398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.12.2.126:43497)
0620 23:09:55.283356 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:55.283367 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:55.283409 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:09:55.285053 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:55.285075 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:55.285514 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:55.285865 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:55.286388 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:55.287740 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:55.289019 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:55.289682 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:55.289855 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:55.290899 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:55.290907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:55.290919 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:55.291470 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:55.291543 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:55.291566 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:55.291669 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:55.291906 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:09:55.292188 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":322,"threads_started":1}
I20260620 23:09:55.294623 32177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b: Bootstrap starting.
I20260620 23:09:55.298478 32177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:55.299775 32177 log.cc:824] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:55.303213 32177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b: No bootstrap required, opened a new log
I20260620 23:09:55.311031 32177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9412bfe985ca469cbd16542d9f2c694b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43497 } }
I20260620 23:09:55.311401 32177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:55.311473 32177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9412bfe985ca469cbd16542d9f2c694b, State: Initialized, Role: FOLLOWER
I20260620 23:09:55.311970 32177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9412bfe985ca469cbd16542d9f2c694b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43497 } }
I20260620 23:09:55.312156 32177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:09:55.312245 32177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:09:55.312372 32177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:55.314416 32177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9412bfe985ca469cbd16542d9f2c694b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43497 } }
I20260620 23:09:55.314867 32177 leader_election.cc:302] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9412bfe985ca469cbd16542d9f2c694b; no voters:
I20260620 23:09:55.315413 32177 leader_election.cc:288] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:09:55.315554 32182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:55.316540 32182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [term 1 LEADER]: Becoming Leader. State: Replica: 9412bfe985ca469cbd16542d9f2c694b, State: Running, Role: LEADER
I20260620 23:09:55.317343 32182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9412bfe985ca469cbd16542d9f2c694b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43497 } }
I20260620 23:09:55.317749 32177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:09:55.320248 32183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9412bfe985ca469cbd16542d9f2c694b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9412bfe985ca469cbd16542d9f2c694b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43497 } } }
I20260620 23:09:55.320236 32184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9412bfe985ca469cbd16542d9f2c694b. Latest consensus state: current_term: 1 leader_uuid: "9412bfe985ca469cbd16542d9f2c694b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9412bfe985ca469cbd16542d9f2c694b" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43497 } } }
I20260620 23:09:55.320768 32183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:55.320768 32184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:55.321635 32191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:09:55.326761 32191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:09:55.335749 32191 catalog_manager.cc:1347] Generated new cluster ID: 270451dced034f2d8503e9ddae83045f
I20260620 23:09:55.335888 32191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:09:55.355875 32191 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:09:55.357635 32191 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:09:55.373274 32191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b: Generated new TSK 0
I20260620 23:09:55.374159 32191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:09:55.390061 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.65:0
--local_ip_for_outbound_sockets=127.12.2.65
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43497
--builtin_ntp_servers=127.12.2.84:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:09:55.720067 32201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:55.720341 32201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:55.720470 32201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:55.727641 32201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:55.727741 32201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:55.727890 32201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.65
I20260620 23:09:55.737047 32201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.2.65
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.65
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:55.738277 32201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:55.740075 32201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:55.752338 32206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:55.752447 32207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:55.752380 32209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:55.753072 32201 server_base.cc:1034] running on GCE node
I20260620 23:09:55.753628 32201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:55.754586 32201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:55.755862 32201 hybrid_clock.cc:648] HybridClock initialized: now 1781996995755785 us; error 54 us; skew 500 ppm
I20260620 23:09:55.756199 32201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:55.758599 32201 webserver.cc:460] Webserver started at http://127.12.2.65:33291/ using document root <none> and password file <none>
I20260620 23:09:55.759368 32201 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:55.759481 32201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:55.759809 32201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:55.762735 32201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/data/instance:
uuid: "740c30a9ed7842c19503536da8e6a799"
format_stamp: "Formatted at 2026-06-20 23:09:55 on dist-test-slave-7g8g"
server_key: "e8f636f495fc0a8d5f7ef5576b98baa7"
server_key_iv: "3c12362c05b004b455fffaf4c04bc812"
server_key_version: "encryptionkey@0"
I20260620 23:09:55.763640 32201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance:
uuid: "740c30a9ed7842c19503536da8e6a799"
format_stamp: "Formatted at 2026-06-20 23:09:55 on dist-test-slave-7g8g"
server_key: "e8f636f495fc0a8d5f7ef5576b98baa7"
server_key_iv: "3c12362c05b004b455fffaf4c04bc812"
server_key_version: "encryptionkey@0"
I20260620 23:09:55.768980 32201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:09:55.772759 32215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:55.774056 32201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 23:09:55.774206 32201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/wal
uuid: "740c30a9ed7842c19503536da8e6a799"
format_stamp: "Formatted at 2026-06-20 23:09:55 on dist-test-slave-7g8g"
server_key: "e8f636f495fc0a8d5f7ef5576b98baa7"
server_key_iv: "3c12362c05b004b455fffaf4c04bc812"
server_key_version: "encryptionkey@0"
I20260620 23:09:55.774407 32201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:55.810225 32201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:55.811069 32201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:55.811295 32201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:55.812168 32201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:55.813572 32201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:55.813654 32201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:55.813731 32201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:55.813791 32201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:55.842655 32201 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.65:44673
I20260620 23:09:55.842705 32327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.65:44673 every 8 connection(s)
I20260620 23:09:55.843928 32201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/data/info.pb
I20260620 23:09:55.846635 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 32201
I20260620 23:09:55.846800 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-0/wal/instance
I20260620 23:09:55.847332 12297 external_mini_cluster.cc:1428] Setting key c2dc1cdebfd620a77554df7d41b2908d
I20260620 23:09:55.850304 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.66:0
--local_ip_for_outbound_sockets=127.12.2.66
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43497
--builtin_ntp_servers=127.12.2.84:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:09:55.857121 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:55.846321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.65:39823 (local address 127.12.2.126:43497)
0620 23:09:55.846837 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:55.846843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:55.848339 (+ 1496us) server_negotiation.cc:407] Connection header received
0620 23:09:55.850239 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:55.850243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:55.850306 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:55.850391 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:55.851757 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:55.852491 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:55.854455 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:55.854871 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:55.855011 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:55.856415 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:55.856420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:55.856423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:55.856442 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:55.856481 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:55.856492 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:55.856572 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:55.856788 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 23:09:55.856870 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":250,"threads_started":1}
I20260620 23:09:55.858403 32331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:55.846917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43497 (local address 127.12.2.65:39823)
0620 23:09:55.847975 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:55.848012 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:55.849958 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:55.850640 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:55.850650 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:55.850761 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:55.851570 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:55.851595 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:55.852635 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:55.852640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:55.854299 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:55.854324 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:55.855087 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:55.855093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:55.855375 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:55.856079 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:09:55.856082 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:55.856130 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:09:55.856230 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:55.856587 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:55.856596 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:55.856604 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:09:55.856769 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 23:09:55.857146 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":274,"threads_started":1}
I20260620 23:09:55.859994 32328 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43497
I20260620 23:09:55.860272 32328 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:55.860980 32328 heartbeater.cc:511] Master 127.12.2.126:43497 requested a full tablet report, sending...
I20260620 23:09:55.863060 32142 ts_manager.cc:194] Registered new tserver with Master: 740c30a9ed7842c19503536da8e6a799 (127.12.2.65:44673)
I20260620 23:09:55.864946 32142 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.65:39823
W20260620 23:09:56.163204 32332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:56.163481 32332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:56.163570 32332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:56.170658 32332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:56.170754 32332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:56.170858 32332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.66
I20260620 23:09:56.179761 32332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.2.66
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.66
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:56.180963 32332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:56.182610 32332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:56.194437 32338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:56.194578 32337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:56.194828 32340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:56.195182 32332 server_base.cc:1034] running on GCE node
I20260620 23:09:56.195700 32332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:56.196549 32332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:56.197731 32332 hybrid_clock.cc:648] HybridClock initialized: now 1781996996197690 us; error 33 us; skew 500 ppm
I20260620 23:09:56.198009 32332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:56.200042 32332 webserver.cc:460] Webserver started at http://127.12.2.66:43819/ using document root <none> and password file <none>
I20260620 23:09:56.200654 32332 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:56.200739 32332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:56.201020 32332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:56.203766 32332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/data/instance:
uuid: "28ad32a01cbd4581966b5421301ea832"
format_stamp: "Formatted at 2026-06-20 23:09:56 on dist-test-slave-7g8g"
server_key: "45c1dc76a91400a2fe9f18bb790b8244"
server_key_iv: "b79b9dc64389668f784f1cb0b5380825"
server_key_version: "encryptionkey@0"
I20260620 23:09:56.204497 32332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance:
uuid: "28ad32a01cbd4581966b5421301ea832"
format_stamp: "Formatted at 2026-06-20 23:09:56 on dist-test-slave-7g8g"
server_key: "45c1dc76a91400a2fe9f18bb790b8244"
server_key_iv: "b79b9dc64389668f784f1cb0b5380825"
server_key_version: "encryptionkey@0"
I20260620 23:09:56.209044 32332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:09:56.212257 32346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:56.213477 32332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:09:56.213603 32332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/wal
uuid: "28ad32a01cbd4581966b5421301ea832"
format_stamp: "Formatted at 2026-06-20 23:09:56 on dist-test-slave-7g8g"
server_key: "45c1dc76a91400a2fe9f18bb790b8244"
server_key_iv: "b79b9dc64389668f784f1cb0b5380825"
server_key_version: "encryptionkey@0"
I20260620 23:09:56.213785 32332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:56.239578 32332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:56.240315 32332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:56.240554 32332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:56.241421 32332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:56.242787 32332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:56.242868 32332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:56.242995 32332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:56.243062 32332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:56.272277 32332 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.66:40357
I20260620 23:09:56.272329 32458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.66:40357 every 8 connection(s)
I20260620 23:09:56.273525 32332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/data/info.pb
I20260620 23:09:56.277084 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 32332
I20260620 23:09:56.277278 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/wal/instance
I20260620 23:09:56.277803 12297 external_mini_cluster.cc:1428] Setting key 6febf65c833e2a88d4b532915321a86e
I20260620 23:09:56.280881 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.2.67:0
--local_ip_for_outbound_sockets=127.12.2.67
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43497
--builtin_ntp_servers=127.12.2.84:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:09:56.286159 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:56.275904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.66:52359 (local address 127.12.2.126:43497)
0620 23:09:56.276079 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:56.276083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:56.277758 (+ 1675us) server_negotiation.cc:407] Connection header received
0620 23:09:56.280110 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:56.280115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:56.280193 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:56.280273 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:56.281361 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.282105 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.283641 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.284065 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.284168 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:56.285402 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:56.285406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:56.285410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:56.285427 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:56.285457 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:56.285468 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:56.285537 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:56.285732 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 23:09:56.285835 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:09:56.287245 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:56.276392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43497 (local address 127.12.2.66:52359)
0620 23:09:56.277589 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:56.277641 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:56.279843 (+ 2202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:56.280505 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:56.280514 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:56.280604 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:56.281209 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:56.281228 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.282246 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.282251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:56.283510 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:56.283522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.284266 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.284273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:56.284493 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:56.285105 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:09:56.285107 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:56.285146 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:09:56.285222 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:56.285550 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:56.285560 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:56.285567 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:09:56.285710 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 23:09:56.286085 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":262,"threads_started":1}
I20260620 23:09:56.288542 32459 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43497
I20260620 23:09:56.288825 32459 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:56.289500 32459 heartbeater.cc:511] Master 127.12.2.126:43497 requested a full tablet report, sending...
I20260620 23:09:56.290835 32142 ts_manager.cc:194] Registered new tserver with Master: 28ad32a01cbd4581966b5421301ea832 (127.12.2.66:40357)
I20260620 23:09:56.291918 32142 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.66:52359
W20260620 23:09:56.591017 32462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:56.591277 32462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:56.591370 32462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:56.598407 32462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:56.598505 32462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:56.598621 32462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.2.67
I20260620 23:09:56.607663 32462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.2.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.2.67
--webserver_port=0
--tserver_master_addrs=127.12.2.126:43497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.2.67
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:56.608825 32462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:56.610472 32462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:56.622911 32467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:56.622996 32470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:56.623011 32468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:56.623739 32462 server_base.cc:1034] running on GCE node
I20260620 23:09:56.624280 32462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:56.625119 32462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:56.626300 32462 hybrid_clock.cc:648] HybridClock initialized: now 1781996996626252 us; error 40 us; skew 500 ppm
I20260620 23:09:56.626593 32462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:09:56.628762 32462 webserver.cc:460] Webserver started at http://127.12.2.67:41287/ using document root <none> and password file <none>
I20260620 23:09:56.629403 32462 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:56.629487 32462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:56.629763 32462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:56.632535 32462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/data/instance:
uuid: "1b22c0c225ee4f4d8e932e304a3e66df"
format_stamp: "Formatted at 2026-06-20 23:09:56 on dist-test-slave-7g8g"
server_key: "a3b829660d19028f2396d3b44cf2b3b3"
server_key_iv: "6013659e96f0c3b869207e0afc98ae57"
server_key_version: "encryptionkey@0"
I20260620 23:09:56.633270 32462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance:
uuid: "1b22c0c225ee4f4d8e932e304a3e66df"
format_stamp: "Formatted at 2026-06-20 23:09:56 on dist-test-slave-7g8g"
server_key: "a3b829660d19028f2396d3b44cf2b3b3"
server_key_iv: "6013659e96f0c3b869207e0afc98ae57"
server_key_version: "encryptionkey@0"
I20260620 23:09:56.638089 32462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:09:56.641292 32476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:56.642488 32462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:09:56.642619 32462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/wal
uuid: "1b22c0c225ee4f4d8e932e304a3e66df"
format_stamp: "Formatted at 2026-06-20 23:09:56 on dist-test-slave-7g8g"
server_key: "a3b829660d19028f2396d3b44cf2b3b3"
server_key_iv: "6013659e96f0c3b869207e0afc98ae57"
server_key_version: "encryptionkey@0"
I20260620 23:09:56.642796 32462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:56.653569 32462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:56.654263 32462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:56.654500 32462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:56.655422 32462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:09:56.656754 32462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:09:56.656854 32462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:56.656955 32462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:09:56.657023 32462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:56.685614 32462 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.67:40435
I20260620 23:09:56.685659 32588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.67:40435 every 8 connection(s)
I20260620 23:09:56.686888 32462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/data/info.pb
I20260620 23:09:56.695560 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 32462
I20260620 23:09:56.695720 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/wal/instance
I20260620 23:09:56.696148 12297 external_mini_cluster.cc:1428] Setting key 8992034c273328a509bcf99e66d89999
I20260620 23:09:56.699127 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:56.689553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:50377 (local address 127.12.2.126:43497)
0620 23:09:56.689728 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:56.689733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:56.691069 (+ 1336us) server_negotiation.cc:407] Connection header received
0620 23:09:56.692794 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:56.692800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:56.692874 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:56.692974 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:56.694003 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.694724 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.696350 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.696861 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.696987 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:56.698303 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:56.698315 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:56.698318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:56.698335 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:56.698367 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:56.698379 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:56.698453 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:56.698655 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 23:09:56.698753 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:09:56.700093 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:56.690038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:43497 (local address 127.12.2.67:50377)
0620 23:09:56.690891 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:56.690964 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:56.692532 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:56.693187 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:56.693196 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:56.693287 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:09:56.693838 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:56.693854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.694858 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.694862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:56.696198 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:56.696210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.697097 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.697105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:56.697338 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:56.698010 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:09:56.698012 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:09:56.698055 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:09:56.698131 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:09:56.698465 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:09:56.698476 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:09:56.698485 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:09:56.698640 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 23:09:56.699052 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":253,"threads_started":1}
I20260620 23:09:56.701200 32589 heartbeater.cc:347] Connected to a master server at 127.12.2.126:43497
I20260620 23:09:56.701465 32589 heartbeater.cc:464] Registering TS with master...
I20260620 23:09:56.702121 32589 heartbeater.cc:511] Master 127.12.2.126:43497 requested a full tablet report, sending...
I20260620 23:09:56.703337 32142 ts_manager.cc:194] Registered new tserver with Master: 1b22c0c225ee4f4d8e932e304a3e66df (127.12.2.67:40435)
I20260620 23:09:56.704077 32142 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.2.67:50377
I20260620 23:09:56.710799 12297 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:09:56.720544 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:56.714726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.12.2.126:43497)
0620 23:09:56.714887 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:56.714893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:56.715293 (+ 400us) server_negotiation.cc:407] Connection header received
0620 23:09:56.715472 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:56.715476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:56.715548 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:56.715636 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:56.716148 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.716906 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.718325 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.718791 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.718896 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:56.719866 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:56.719871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:56.719874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:56.719891 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:56.719922 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:56.719934 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:56.720006 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:56.720147 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 23:09:56.720239 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:09:56.724328 32142 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:09:56.731016 32142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:09:56.763305 32602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:56.742505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.67:40435 (local address 127.0.0.1:46074)
0620 23:09:56.754427 (+ 11922us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:56.754483 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:56.754691 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:56.756338 (+ 1647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:56.756343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:56.756372 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:56.756758 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:56.756771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.758736 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.758741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:56.760676 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:56.760687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.762402 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.762409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:56.762599 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:56.762648 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:09:56.762780 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:09:56.762994 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11614,"thread_start_us":12365,"threads_started":1}
I20260620 23:09:56.763610 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:56.743114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.12.2.66:40357)
0620 23:09:56.754727 (+ 11613us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:56.754735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:56.755284 (+ 549us) server_negotiation.cc:407] Connection header received
0620 23:09:56.755599 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:56.755608 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:56.755844 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:56.756007 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:56.757112 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.758470 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.760910 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.762219 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.762349 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:56.762615 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:56.763113 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 23:09:56.763274 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11382,"thread_start_us":11266,"threads_started":1}
I20260620 23:09:56.763681 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:56.743086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.12.2.67:40435)
0620 23:09:56.754045 (+ 10959us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:56.754057 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:56.754592 (+ 535us) server_negotiation.cc:407] Connection header received
0620 23:09:56.755037 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:56.755047 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:56.755304 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:56.755471 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:56.756936 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.758592 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.760828 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.762218 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.762349 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:56.762615 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:56.762842 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 23:09:56.763046 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10749,"thread_start_us":10976,"threads_started":1}
I20260620 23:09:56.764346 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:56.755453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:44673 (local address 127.0.0.1:57378)
0620 23:09:56.757271 (+ 1818us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:56.757287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:56.757405 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:56.757921 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:56.757925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:56.757955 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:56.758336 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:56.758348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.760140 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.760145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:56.761645 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:56.761656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.763622 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.763629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:56.763788 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:56.763819 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:09:56.763903 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:09:56.764018 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1711,"thread_start_us":1579,"threads_started":1}
I20260620 23:09:56.764750 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:56.755701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.12.2.65:44673)
0620 23:09:56.756150 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:56.756157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:56.757400 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 23:09:56.757530 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:56.757537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:56.757734 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:56.757916 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:56.758500 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.759995 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.761804 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.762915 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.763060 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:56.763278 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:56.764301 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0620 23:09:56.764453 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":232,"threads_started":1}
I20260620 23:09:56.765053 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:56.742604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:40357 (local address 127.0.0.1:46380)
0620 23:09:56.755171 (+ 12567us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:56.755188 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:56.755415 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:56.756379 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:56.756385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:56.756417 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:56.756899 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:56.756910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.758615 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.758619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:56.760729 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:56.760743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:56.762738 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:56.762742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:56.762894 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:56.762924 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:09:56.763859 (+ 935us) client_negotiation.cc:240] Negotiation successful
0620 23:09:56.763997 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12432,"mutex_wait_us":90,"thread_start_us":11978,"threads_started":1}
I20260620 23:09:56.767903 32394 tablet_service.cc:1467] Processing CreateTablet for tablet 78845ed8b35a4ce99d9b78d48411e3ad (DEFAULT_TABLE table=test-table [id=b71ee88a3334463ea5b4edbf5d79eaad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:56.767736 32524 tablet_service.cc:1467] Processing CreateTablet for tablet 78845ed8b35a4ce99d9b78d48411e3ad (DEFAULT_TABLE table=test-table [id=b71ee88a3334463ea5b4edbf5d79eaad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:56.768146 32263 tablet_service.cc:1467] Processing CreateTablet for tablet 78845ed8b35a4ce99d9b78d48411e3ad (DEFAULT_TABLE table=test-table [id=b71ee88a3334463ea5b4edbf5d79eaad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:09:56.769140 32394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78845ed8b35a4ce99d9b78d48411e3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:56.769210 32263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78845ed8b35a4ce99d9b78d48411e3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:56.769209 32524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78845ed8b35a4ce99d9b78d48411e3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:56.781109 32607 tablet_bootstrap.cc:492] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: Bootstrap starting.
I20260620 23:09:56.782860 32608 tablet_bootstrap.cc:492] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: Bootstrap starting.
I20260620 23:09:56.783600 32609 tablet_bootstrap.cc:492] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799: Bootstrap starting.
I20260620 23:09:56.784358 32607 tablet_bootstrap.cc:654] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:56.785578 32607 log.cc:824] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:56.786599 32608 tablet_bootstrap.cc:654] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:56.786693 32609 tablet_bootstrap.cc:654] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:56.787567 32608 log.cc:824] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:56.787858 32609 log.cc:824] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:56.788712 32607 tablet_bootstrap.cc:492] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: No bootstrap required, opened a new log
I20260620 23:09:56.788946 32607 ts_tablet_manager.cc:1397] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260620 23:09:56.790349 32608 tablet_bootstrap.cc:492] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: No bootstrap required, opened a new log
I20260620 23:09:56.790596 32608 ts_tablet_manager.cc:1397] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 23:09:56.792457 32609 tablet_bootstrap.cc:492] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799: No bootstrap required, opened a new log
I20260620 23:09:56.792716 32609 ts_tablet_manager.cc:1397] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:09:56.797485 32608 raft_consensus.cc:348] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40435 } } peers { permanent_uuid: "28ad32a01cbd4581966b5421301ea832" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40357 } } peers { permanent_uuid: "740c30a9ed7842c19503536da8e6a799" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 44673 } }
I20260620 23:09:56.797861 32608 raft_consensus.cc:374] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:56.797964 32608 raft_consensus.cc:729] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b22c0c225ee4f4d8e932e304a3e66df, State: Initialized, Role: FOLLOWER
I20260620 23:09:56.798535 32608 consensus_queue.cc:260] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40435 } } peers { permanent_uuid: "28ad32a01cbd4581966b5421301ea832" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40357 } } peers { permanent_uuid: "740c30a9ed7842c19503536da8e6a799" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 44673 } }
I20260620 23:09:56.798517 32607 raft_consensus.cc:348] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40435 } } peers { permanent_uuid: "28ad32a01cbd4581966b5421301ea832" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40357 } } peers { permanent_uuid: "740c30a9ed7842c19503536da8e6a799" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 44673 } }
I20260620 23:09:56.798970 32607 raft_consensus.cc:374] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:56.799067 32607 raft_consensus.cc:729] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ad32a01cbd4581966b5421301ea832, State: Initialized, Role: FOLLOWER
I20260620 23:09:56.799647 32589 heartbeater.cc:503] Master 127.12.2.126:43497 was elected leader, sending a full tablet report...
I20260620 23:09:56.799710 32607 consensus_queue.cc:260] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40435 } } peers { permanent_uuid: "28ad32a01cbd4581966b5421301ea832" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40357 } } peers { permanent_uuid: "740c30a9ed7842c19503536da8e6a799" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 44673 } }
I20260620 23:09:56.800282 32608 ts_tablet_manager.cc:1428] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 23:09:56.801277 32459 heartbeater.cc:503] Master 127.12.2.126:43497 was elected leader, sending a full tablet report...
I20260620 23:09:56.801882 32607 ts_tablet_manager.cc:1428] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260620 23:09:56.802743 32609 raft_consensus.cc:348] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40435 } } peers { permanent_uuid: "28ad32a01cbd4581966b5421301ea832" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40357 } } peers { permanent_uuid: "740c30a9ed7842c19503536da8e6a799" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 44673 } }
I20260620 23:09:56.803189 32609 raft_consensus.cc:374] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:56.803313 32609 raft_consensus.cc:729] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 740c30a9ed7842c19503536da8e6a799, State: Initialized, Role: FOLLOWER
I20260620 23:09:56.803891 32609 consensus_queue.cc:260] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40435 } } peers { permanent_uuid: "28ad32a01cbd4581966b5421301ea832" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40357 } } peers { permanent_uuid: "740c30a9ed7842c19503536da8e6a799" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 44673 } }
I20260620 23:09:56.805584 32328 heartbeater.cc:503] Master 127.12.2.126:43497 was elected leader, sending a full tablet report...
I20260620 23:09:56.806239 32609 ts_tablet_manager.cc:1428] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260620 23:09:56.845876 32329 tablet.cc:2369] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:09:56.938589 32590 tablet.cc:2369] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:09:57.022110 32613 raft_consensus.cc:482] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:09:57.022384 32613 raft_consensus.cc:504] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40435 } } peers { permanent_uuid: "28ad32a01cbd4581966b5421301ea832" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40357 } } peers { permanent_uuid: "740c30a9ed7842c19503536da8e6a799" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 44673 } }
I20260620 23:09:57.023999 32613 leader_election.cc:288] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28ad32a01cbd4581966b5421301ea832 (127.12.2.66:40357), 740c30a9ed7842c19503536da8e6a799 (127.12.2.65:44673)
W20260620 23:09:57.025333 32460 tablet.cc:2369] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:09:57.029548 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.024391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:36003 (local address 127.12.2.66:40357)
0620 23:09:57.024597 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:57.024603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:57.024625 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:09:57.024788 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:57.024794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:57.024874 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:57.024976 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:57.025415 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.026382 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.028021 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.028758 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.028866 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:57.028969 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:57.029196 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 23:09:57.029313 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:09:57.029596 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.024359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.66:40357 (local address 127.12.2.67:36003)
0620 23:09:57.024539 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:57.024556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:57.024659 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:57.025026 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:57.025031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:57.025061 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:57.025269 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:57.025279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.026529 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.026533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:57.027860 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:57.027872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.028890 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.028896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:57.029048 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:57.029071 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:09:57.029173 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:09:57.029313 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 23:09:57.030474 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.024792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:44673 (local address 127.12.2.67:42461)
0620 23:09:57.025548 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:57.025564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:09:57.025688 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:09:57.026074 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:09:57.026078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:09:57.026105 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:09:57.026408 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:57.026418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.027436 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.027440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:57.028702 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:09:57.028714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.029940 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.029946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:09:57.030069 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:57.030090 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:09:57.030174 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:09:57.030253 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":965,"threads_started":1}
I20260620 23:09:57.030573 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.025027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.2.67:42461 (local address 127.12.2.65:44673)
0620 23:09:57.025208 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:57.025213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:57.025655 (+ 442us) server_negotiation.cc:407] Connection header received
0620 23:09:57.025813 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:57.025819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:57.025905 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:57.026046 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:09:57.026578 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.027292 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.028883 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.029708 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.029844 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:57.029939 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:57.030219 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 23:09:57.030338 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:09:57.030441 32414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78845ed8b35a4ce99d9b78d48411e3ad" candidate_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ad32a01cbd4581966b5421301ea832" is_pre_election: true
I20260620 23:09:57.031002 32414 raft_consensus.cc:2454] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b22c0c225ee4f4d8e932e304a3e66df in term 0.
I20260620 23:09:57.031597 32478 leader_election.cc:302] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b22c0c225ee4f4d8e932e304a3e66df, 28ad32a01cbd4581966b5421301ea832; no voters:
I20260620 23:09:57.031589 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78845ed8b35a4ce99d9b78d48411e3ad" candidate_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "740c30a9ed7842c19503536da8e6a799" is_pre_election: true
I20260620 23:09:57.031935 32613 raft_consensus.cc:2781] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:09:57.032049 32613 raft_consensus.cc:482] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:09:57.032055 32283 raft_consensus.cc:2454] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b22c0c225ee4f4d8e932e304a3e66df in term 0.
I20260620 23:09:57.032137 32613 raft_consensus.cc:3037] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:57.034637 32613 raft_consensus.cc:504] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40435 } } peers { permanent_uuid: "28ad32a01cbd4581966b5421301ea832" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40357 } } peers { permanent_uuid: "740c30a9ed7842c19503536da8e6a799" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 44673 } }
I20260620 23:09:57.035216 32613 leader_election.cc:288] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [CANDIDATE]: Term 1 election: Requested vote from peers 28ad32a01cbd4581966b5421301ea832 (127.12.2.66:40357), 740c30a9ed7842c19503536da8e6a799 (127.12.2.65:44673)
I20260620 23:09:57.035597 32414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78845ed8b35a4ce99d9b78d48411e3ad" candidate_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ad32a01cbd4581966b5421301ea832"
I20260620 23:09:57.035683 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78845ed8b35a4ce99d9b78d48411e3ad" candidate_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "740c30a9ed7842c19503536da8e6a799"
I20260620 23:09:57.035772 32414 raft_consensus.cc:3037] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:57.035923 32283 raft_consensus.cc:3037] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:57.037791 32414 raft_consensus.cc:2454] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b22c0c225ee4f4d8e932e304a3e66df in term 1.
I20260620 23:09:57.038034 32283 raft_consensus.cc:2454] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b22c0c225ee4f4d8e932e304a3e66df in term 1.
I20260620 23:09:57.038080 32478 leader_election.cc:302] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b22c0c225ee4f4d8e932e304a3e66df, 28ad32a01cbd4581966b5421301ea832; no voters:
I20260620 23:09:57.038308 32613 raft_consensus.cc:2781] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:57.038707 32613 raft_consensus.cc:686] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 1 LEADER]: Becoming Leader. State: Replica: 1b22c0c225ee4f4d8e932e304a3e66df, State: Running, Role: LEADER
I20260620 23:09:57.039212 32613 consensus_queue.cc:237] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40435 } } peers { permanent_uuid: "28ad32a01cbd4581966b5421301ea832" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40357 } } peers { permanent_uuid: "740c30a9ed7842c19503536da8e6a799" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 44673 } }
I20260620 23:09:57.043243 32142 catalog_manager.cc:5495] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b22c0c225ee4f4d8e932e304a3e66df (127.12.2.67). New cstate: current_term: 1 leader_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b22c0c225ee4f4d8e932e304a3e66df" member_type: VOTER last_known_addr { host: "127.12.2.67" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28ad32a01cbd4581966b5421301ea832" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "740c30a9ed7842c19503536da8e6a799" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 44673 } health_report { overall_health: UNKNOWN } } }
I20260620 23:09:57.116025 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.111162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.12.2.67:40435)
0620 23:09:57.111381 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:57.111386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:57.111403 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:09:57.111581 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:57.111585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:57.111661 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:57.111778 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:09:57.112063 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.112817 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.114167 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.114577 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.114669 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:57.115078 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:09:57.115571 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:09:57.115676 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:57.115733 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 23:09:57.115794 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:09:57.123874 32283 raft_consensus.cc:1261] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799 [term 1 FOLLOWER]: Refusing update from remote peer 1b22c0c225ee4f4d8e932e304a3e66df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:57.123965 32414 raft_consensus.cc:1261] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832 [term 1 FOLLOWER]: Refusing update from remote peer 1b22c0c225ee4f4d8e932e304a3e66df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:09:57.124733 32617 consensus_queue.cc:1035] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [LEADER]: Connected to new peer: Peer: permanent_uuid: "740c30a9ed7842c19503536da8e6a799" member_type: VOTER last_known_addr { host: "127.12.2.65" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:09:57.125123 32613 consensus_queue.cc:1035] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [LEADER]: Connected to new peer: Peer: permanent_uuid: "28ad32a01cbd4581966b5421301ea832" member_type: VOTER last_known_addr { host: "127.12.2.66" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:09:57.138624 32621 mvcc.cc:205] Tried to move back new op lower bound from 7299059700209664000 to 7299059699876261888. Current Snapshot: MvccSnapshot[applied={T|T < 7299059700209664000}]
I20260620 23:09:57.147532 32142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:57.147802 32142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:09:57.152202 32142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9412bfe985ca469cbd16542d9f2c694b: Sending DeleteTablet for 3 replicas of tablet 78845ed8b35a4ce99d9b78d48411e3ad
I20260620 23:09:57.153149 32394 tablet_service.cc:1514] Processing DeleteTablet for tablet 78845ed8b35a4ce99d9b78d48411e3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:57 UTC) from {username='slave'} at 127.0.0.1:46380
I20260620 23:09:57.153215 32524 tablet_service.cc:1514] Processing DeleteTablet for tablet 78845ed8b35a4ce99d9b78d48411e3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:57 UTC) from {username='slave'} at 127.0.0.1:46074
I20260620 23:09:57.153797 32393 tablet_service.cc:1514] Processing DeleteTablet for tablet 78845ed8b35a4ce99d9b78d48411e3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:57 UTC) from {username='slave'} at 127.0.0.1:46380
I20260620 23:09:57.153990 32263 tablet_service.cc:1514] Processing DeleteTablet for tablet 78845ed8b35a4ce99d9b78d48411e3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:09:57 UTC) from {username='slave'} at 127.0.0.1:57378
I20260620 23:09:57.157739 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 32201
I20260620 23:09:57.163757 32631 tablet_replica.cc:330] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: stopping tablet replica
I20260620 23:09:57.164333 32631 raft_consensus.cc:2229] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:09:57.165067 32631 raft_consensus.cc:2258] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:57.167287 32631 ts_tablet_manager.cc:1905] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:57.172140 32632 tablet_replica.cc:330] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799: stopping tablet replica
I20260620 23:09:57.172603 32632 raft_consensus.cc:2229] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:57.173045 32630 tablet_replica.cc:330] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: stopping tablet replica
I20260620 23:09:57.173086 32632 raft_consensus.cc:2258] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:57.173445 32630 raft_consensus.cc:2229] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:09:57.173918 32630 raft_consensus.cc:2258] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:09:57.175231 32632 ts_tablet_manager.cc:1905] T 78845ed8b35a4ce99d9b78d48411e3ad P 740c30a9ed7842c19503536da8e6a799: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:57.175853 32630 ts_tablet_manager.cc:1905] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:09:57.176249 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.159035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.12.2.65:44673)
0620 23:09:57.159223 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:57.159230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:57.166119 (+ 6889us) server_negotiation.cc:407] Connection header received
0620 23:09:57.166390 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:57.166397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:57.166538 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:57.166620 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:57.167359 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.168652 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.170602 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.171333 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.171430 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:57.174340 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:57.174351 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:57.174368 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:57.175146 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:57.175206 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:57.175234 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:57.175318 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:57.175596 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 23:09:57.175770 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:09:57.177021 32631 ts_tablet_manager.cc:1918] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:57.177168 32631 log.cc:1196] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-2/wal/wals/78845ed8b35a4ce99d9b78d48411e3ad
I20260620 23:09:57.177464 32323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:57.177686 32631 ts_tablet_manager.cc:1939] T 78845ed8b35a4ce99d9b78d48411e3ad P 1b22c0c225ee4f4d8e932e304a3e66df: Deleting consensus metadata
I20260620 23:09:57.179628 32128 catalog_manager.cc:4841] TS 1b22c0c225ee4f4d8e932e304a3e66df (127.12.2.67:40435): tablet 78845ed8b35a4ce99d9b78d48411e3ad (table test-table [id=b71ee88a3334463ea5b4edbf5d79eaad]) successfully deleted
I20260620 23:09:57.183104 32630 ts_tablet_manager.cc:1918] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:09:57.183324 32630 log.cc:1196] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: Deleting WAL directory at /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996696601630-12297-0/minicluster-data/ts-1/wal/wals/78845ed8b35a4ce99d9b78d48411e3ad
I20260620 23:09:57.183869 32630 ts_tablet_manager.cc:1939] T 78845ed8b35a4ce99d9b78d48411e3ad P 28ad32a01cbd4581966b5421301ea832: Deleting consensus metadata
I20260620 23:09:57.185647 32127 catalog_manager.cc:4841] TS 28ad32a01cbd4581966b5421301ea832 (127.12.2.66:40357): tablet 78845ed8b35a4ce99d9b78d48411e3ad (table test-table [id=b71ee88a3334463ea5b4edbf5d79eaad]) successfully deleted
W20260620 23:09:57.186093 32127 catalog_manager.cc:4808] TS 28ad32a01cbd4581966b5421301ea832 (127.12.2.66:40357): delete failed for tablet 78845ed8b35a4ce99d9b78d48411e3ad because the tablet was not found. No further retry: Not found: Tablet not found: 78845ed8b35a4ce99d9b78d48411e3ad
I20260620 23:09:57.267988 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 32201
W20260620 23:09:57.285110 32127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.2.65:44673 (error 108)
I20260620 23:09:57.291522 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 32332
I20260620 23:09:57.305378 32636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.304167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:44673 (local address 127.0.0.1:57384)
0620 23:09:57.304949 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:57.305093 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:44673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":566,"threads_started":1}
W20260620 23:09:57.305766 32127 catalog_manager.cc:4584] TS 740c30a9ed7842c19503536da8e6a799 (127.12.2.65:44673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78845ed8b35a4ce99d9b78d48411e3ad: Network error: Client connection negotiation failed: client connection to 127.12.2.65:44673: connect: Connection refused (error 111)
I20260620 23:09:57.307020 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.293044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.12.2.66:40357)
0620 23:09:57.293206 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:57.293212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:57.293230 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:09:57.293298 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:57.293303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:57.293417 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:57.293505 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:57.294077 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.302162 (+ 8085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.303718 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.304303 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.304440 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:57.305683 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:57.305692 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:57.305705 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:57.306317 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:57.306367 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:57.306386 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:57.306475 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:57.306632 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 23:09:57.306715 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:09:57.307698 32454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:57.359594 32636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.359051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:44673 (local address 127.0.0.1:57386)
0620 23:09:57.359307 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:57.359421 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:44673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:09:57.360733 32636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.360340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:44673 (local address 127.0.0.1:57392)
0620 23:09:57.360489 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:57.360575 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:44673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 23:09:57.394661 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 32332
I20260620 23:09:57.408228 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 32462
I20260620 23:09:57.416098 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.408917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.12.2.67:40435)
0620 23:09:57.409079 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:57.409087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:57.409292 (+ 205us) server_negotiation.cc:407] Connection header received
0620 23:09:57.409479 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:57.409482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:57.409560 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:57.409651 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:57.410117 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.410898 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.412671 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:57.413186 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:57.413276 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:57.414546 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:57.414556 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:57.414570 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:09:57.415288 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:57.415340 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:09:57.415374 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:57.415465 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:57.415622 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 23:09:57.415705 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:09:57.416872 32584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:57.421115 32636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.420625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:44673 (local address 127.0.0.1:57394)
0620 23:09:57.420841 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:57.420941 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:44673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 23:09:57.422223 32636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.421825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:44673 (local address 127.0.0.1:57406)
0620 23:09:57.421975 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:57.422053 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:44673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:09:57.503247 32636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.502608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:44673 (local address 127.0.0.1:57416)
0620 23:09:57.502870 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:57.503020 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:44673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 23:09:57.504463 32636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:57.503922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.65:44673 (local address 127.0.0.1:57430)
0620 23:09:57.504170 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 23:09:57.504283 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.65:44673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 23:09:57.549266 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 32462
I20260620 23:09:57.564371 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 32111
I20260620 23:09:57.565428 32172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:57.671589 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 32111
2026-06-20T23:09:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2877 ms)
[----------] 6 tests from EncryptionPolicyTest (20146 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:09:57 dist-test-slave-7g8g krb5kdc[32642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:09:57 dist-test-slave-7g8g krb5kdc[32642](info): set up 2 sockets
Jun 20 23:09:57 dist-test-slave-7g8g krb5kdc[32642](info): commencing operation
krb5kdc: starting...
W20260620 23:09:58.687055 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:09:58 dist-test-slave-7g8g krb5kdc[32642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:09:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:09:58.912285 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:09:59.224036 32655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:09:59.224298 32655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:09:59.224416 32655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:09:59.231398 32655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:09:59.231509 32655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:09:59.231578 32655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:09:59.231657 32655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:09:59.231716 32655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:09:59.241359 32655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:09:59.242556 32655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:09:59.243795 32655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:09:59.254432 32661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:59.254503 32660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:09:59.254477 32663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:09:59.255584 32655 server_base.cc:1034] running on GCE node
I20260620 23:09:59.256335 32655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:09:59.257608 32655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:09:59.258812 32655 hybrid_clock.cc:648] HybridClock initialized: now 1781996999258774 us; error 24 us; skew 500 ppm
I20260620 23:09:59.259160 32655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:09:59 dist-test-slave-7g8g krb5kdc[32642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:09:59.264940 32655 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:09:59.266268 32655 webserver.cc:460] Webserver started at http://127.0.0.1:40379/ using document root <none> and password file <none>
I20260620 23:09:59.266891 32655 fs_manager.cc:359] Metadata directory not provided
I20260620 23:09:59.267030 32655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:09:59.267310 32655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:09:59.270081 32655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "db44207478bd4d9283ac7d6b7213e9e9"
format_stamp: "Formatted at 2026-06-20 23:09:59 on dist-test-slave-7g8g"
server_key: "de56dff215c773aba2d90c1e75eb5541"
server_key_iv: "01a2b5d2072a9d342f9ff05e5222c440"
server_key_version: "encryptionkey@0"
I20260620 23:09:59.270800 32655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "db44207478bd4d9283ac7d6b7213e9e9"
format_stamp: "Formatted at 2026-06-20 23:09:59 on dist-test-slave-7g8g"
server_key: "de56dff215c773aba2d90c1e75eb5541"
server_key_iv: "01a2b5d2072a9d342f9ff05e5222c440"
server_key_version: "encryptionkey@0"
I20260620 23:09:59.275548 32655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:09:59.278841 32670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:09:59.280133 32655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:09:59.280283 32655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "db44207478bd4d9283ac7d6b7213e9e9"
format_stamp: "Formatted at 2026-06-20 23:09:59 on dist-test-slave-7g8g"
server_key: "de56dff215c773aba2d90c1e75eb5541"
server_key_iv: "01a2b5d2072a9d342f9ff05e5222c440"
server_key_version: "encryptionkey@0"
I20260620 23:09:59.280460 32655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:09:59.311419 32655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:09:59.315284 32655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:09:59.315550 32655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:09:59.332216 32655 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34203
I20260620 23:09:59.332235 32721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34203 every 8 connection(s)
I20260620 23:09:59.333585 32655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:09:59.337189 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:09:59.342422 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 32655
I20260620 23:09:59.342593 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:09:59.343232 12297 external_mini_cluster.cc:1428] Setting key f47cf5d83fed598188f326345fc17f6b
I20260620 23:09:59.347604 32722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9: Bootstrap starting.
I20260620 23:09:59.351161 32722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9: Neither blocks nor log segments found. Creating new log.
I20260620 23:09:59.352355 32722 log.cc:824] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:09:59.355451 32722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9: No bootstrap required, opened a new log
Jun 20 23:09:59 dist-test-slave-7g8g krb5kdc[32642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:09:59.363508 32722 raft_consensus.cc:348] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db44207478bd4d9283ac7d6b7213e9e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 23:09:59.363860 32722 raft_consensus.cc:374] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:09:59.363976 32722 raft_consensus.cc:729] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db44207478bd4d9283ac7d6b7213e9e9, State: Initialized, Role: FOLLOWER
I20260620 23:09:59.364540 32722 consensus_queue.cc:260] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db44207478bd4d9283ac7d6b7213e9e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 23:09:59.364926 32722 raft_consensus.cc:388] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:09:59.365058 32722 raft_consensus.cc:482] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:09:59.365262 32722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:09:59.366220 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:59.345115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49491 (local address 127.0.0.1:34203)
0620 23:09:59.345885 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:59.345896 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:59.345935 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:09:59.347583 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:59.347609 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:59.348000 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:59.348303 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:59.350065 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:59.351411 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:59.353180 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:59.353987 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:59.354109 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:59.358707 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:59.358714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:59.358733 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:59.358771 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:59.361800 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:59.362596 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:59.362604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:59.362610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:59.362776 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:59.363239 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:59.363243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:59.363245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:59.363764 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:59.364089 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:59.364669 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 23:09:59.364958 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":353,"threads_started":1}
I20260620 23:09:59.367717 32722 raft_consensus.cc:504] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db44207478bd4d9283ac7d6b7213e9e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 23:09:59.368219 32722 leader_election.cc:302] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db44207478bd4d9283ac7d6b7213e9e9; no voters:
I20260620 23:09:59.368810 32722 leader_election.cc:288] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:09:59.368891 32727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:09:59.370000 32727 raft_consensus.cc:686] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [term 1 LEADER]: Becoming Leader. State: Replica: db44207478bd4d9283ac7d6b7213e9e9, State: Running, Role: LEADER
I20260620 23:09:59.370414 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:09:59.370561 32727 consensus_queue.cc:237] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db44207478bd4d9283ac7d6b7213e9e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 23:09:59.370975 32722 sys_catalog.cc:564] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:09:59.377485 32728 sys_catalog.cc:455] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db44207478bd4d9283ac7d6b7213e9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db44207478bd4d9283ac7d6b7213e9e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } } }
I20260620 23:09:59.377453 32729 sys_catalog.cc:455] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db44207478bd4d9283ac7d6b7213e9e9. Latest consensus state: current_term: 1 leader_uuid: "db44207478bd4d9283ac7d6b7213e9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db44207478bd4d9283ac7d6b7213e9e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } } }
I20260620 23:09:59.377915 32728 sys_catalog.cc:458] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:59.377918 32729 sys_catalog.cc:458] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:09:59.379140 32736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:09:59.384959 32736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:09:59.401423 32736 catalog_manager.cc:1347] Generated new cluster ID: 7fd360da759743ea810572766133b285
I20260620 23:09:59.401546 32736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:09:59.417407 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:09:59.393166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46631 (local address 127.0.0.1:34203)
0620 23:09:59.393319 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:09:59.393325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:09:59.399897 (+ 6572us) server_negotiation.cc:407] Connection header received
0620 23:09:59.400178 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:09:59.400184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:09:59.400267 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:09:59.400339 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:09:59.402580 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:59.403801 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:59.405634 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:09:59.406275 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:09:59.406370 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:09:59.409713 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:09:59.409720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:09:59.409726 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:09:59.409763 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:09:59.413223 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:59.414178 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:59.414183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:59.414187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:59.414356 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:09:59.414922 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:09:59.414947 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:09:59.414951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:09:59.415320 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:09:59.415586 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:09:59.416854 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0620 23:09:59.417032 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:09:59.419696 32736 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:09:59.421788 32736 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:09:59.435797 32736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db44207478bd4d9283ac7d6b7213e9e9: Generated new TSK 0
I20260620 23:09:59.436962 32736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:09:59.447754 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 32655
I20260620 23:09:59.449239 32717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:09:59.539521 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 32655
W20260620 23:09:59.554688 32640 connection.cc:535] client connection to 127.0.0.1:34203 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34203): Connection reset by peer (error 104)
2026-06-20T23:09:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1865 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:09:59 dist-test-slave-7g8g krb5kdc[32754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:09:59 dist-test-slave-7g8g krb5kdc[32754](info): set up 2 sockets
Jun 20 23:09:59 dist-test-slave-7g8g krb5kdc[32754](info): commencing operation
krb5kdc: starting...
W20260620 23:10:00.551016 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:00 dist-test-slave-7g8g krb5kdc[32754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:10:00.779659 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:01.088457 32767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:01.088742 32767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:01.088877 32767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:01.096038 32767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:01.096160 32767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:01.096235 32767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:01.096316 32767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:01.096372 32767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:01.106037 32767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:01.107285 32767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:01.108585 32767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:01.119338 307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:01.119374 305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:01.119828 304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:01.120147 32767 server_base.cc:1034] running on GCE node
I20260620 23:10:01.120841 32767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:01.122100 32767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:01.123296 32767 hybrid_clock.cc:648] HybridClock initialized: now 1781997001123248 us; error 36 us; skew 500 ppm
I20260620 23:10:01.123592 32767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:01 dist-test-slave-7g8g krb5kdc[32754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:01.126382 32767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:10:01.127732 32767 webserver.cc:460] Webserver started at http://127.0.0.1:40099/ using document root <none> and password file <none>
I20260620 23:10:01.128331 32767 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:01.128445 32767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:01.128715 32767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:01.131556 32767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "3d189bdd17124db5b2563e219f6f6cc9"
format_stamp: "Formatted at 2026-06-20 23:10:01 on dist-test-slave-7g8g"
server_key: "42d2823c71d03b18bc59c6d16e9b86ab"
server_key_iv: "b64693cbc9a45093659414179071577c"
server_key_version: "encryptionkey@0"
I20260620 23:10:01.132248 32767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "3d189bdd17124db5b2563e219f6f6cc9"
format_stamp: "Formatted at 2026-06-20 23:10:01 on dist-test-slave-7g8g"
server_key: "42d2823c71d03b18bc59c6d16e9b86ab"
server_key_iv: "b64693cbc9a45093659414179071577c"
server_key_version: "encryptionkey@0"
I20260620 23:10:01.136974 32767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:10:01.140130 314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:01.141309 32767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:10:01.141481 32767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "3d189bdd17124db5b2563e219f6f6cc9"
format_stamp: "Formatted at 2026-06-20 23:10:01 on dist-test-slave-7g8g"
server_key: "42d2823c71d03b18bc59c6d16e9b86ab"
server_key_iv: "b64693cbc9a45093659414179071577c"
server_key_version: "encryptionkey@0"
I20260620 23:10:01.141646 32767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:01.165284 32767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:01.168897 32767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:01.169166 32767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:01.185438 32767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34795
I20260620 23:10:01.185467 365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34795 every 8 connection(s)
I20260620 23:10:01.186685 32767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:01.189639 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 32767
I20260620 23:10:01.189801 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:01.190236 12297 external_mini_cluster.cc:1428] Setting key 68f8a8165bfa11329673ecfb44b1ac81
I20260620 23:10:01.190286 366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:01.203272 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:01.191819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60313 (local address 127.0.0.1:34795)
0620 23:10:01.192530 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:01.192541 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:01.192580 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:10:01.194614 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:01.194635 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:01.195021 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:01.195320 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:01.195875 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:01.197166 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:01.199041 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:01.199668 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:01.199792 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:01.201053 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:01.201063 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:01.201075 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:01.201619 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:01.201676 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:01.201697 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:01.201802 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:01.202036 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 23:10:01.202190 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":321,"threads_started":1}
I20260620 23:10:01.203359 366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9: Bootstrap starting.
I20260620 23:10:01.206645 366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:01.206954 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:01.207846 366 log.cc:824] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:01.210768 366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9: No bootstrap required, opened a new log
I20260620 23:10:01.217764 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:01.211485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58023 (local address 127.0.0.1:34795)
0620 23:10:01.211645 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:01.211650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:01.212085 (+ 435us) server_negotiation.cc:407] Connection header received
0620 23:10:01.212212 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:01.212217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:01.212303 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:01.212400 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:01.212896 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:01.214014 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:01.215292 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:01.216004 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:01.216132 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:01.217020 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:01.217026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:01.217030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:01.217055 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:01.217124 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:01.217144 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:01.217281 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:01.217402 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 23:10:01.217451 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:10:01.219718 366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d189bdd17124db5b2563e219f6f6cc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } }
I20260620 23:10:01.220048 366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:01.220119 366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d189bdd17124db5b2563e219f6f6cc9, State: Initialized, Role: FOLLOWER
I20260620 23:10:01.220615 366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d189bdd17124db5b2563e219f6f6cc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } }
I20260620 23:10:01.220824 366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:01.220913 366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:01.221030 366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:01.223070 366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d189bdd17124db5b2563e219f6f6cc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } }
I20260620 23:10:01.223505 366 leader_election.cc:302] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d189bdd17124db5b2563e219f6f6cc9; no voters:
I20260620 23:10:01.223980 366 leader_election.cc:288] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:01.224087 376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:01.225114 376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 3d189bdd17124db5b2563e219f6f6cc9, State: Running, Role: LEADER
I20260620 23:10:01.225687 376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d189bdd17124db5b2563e219f6f6cc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } }
I20260620 23:10:01.225968 366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:01.228003 378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d189bdd17124db5b2563e219f6f6cc9. Latest consensus state: current_term: 1 leader_uuid: "3d189bdd17124db5b2563e219f6f6cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d189bdd17124db5b2563e219f6f6cc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } } }
I20260620 23:10:01.228348 378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:01.228962 377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d189bdd17124db5b2563e219f6f6cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d189bdd17124db5b2563e219f6f6cc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } } }
I20260620 23:10:01.229259 377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:01.231900 383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:01.236212 383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:01.243678 383 catalog_manager.cc:1347] Generated new cluster ID: c7846a4d71284772835646c477451601
I20260620 23:10:01.243832 383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:01.259829 383 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:01.260998 383 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:01.278466 383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d189bdd17124db5b2563e219f6f6cc9: Generated new TSK 0
I20260620 23:10:01.279280 383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:01.291599 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 32767
W20260620 23:10:01.292889 361 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60313
W20260620 23:10:01.293552 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 32767: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:01.293705 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 32767
W20260620 23:10:01.310212 32752 connection.cc:535] client connection to 127.0.0.1:34795 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34795): Connection reset by peer (error 104)
2026-06-20T23:10:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1754 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:01 dist-test-slave-7g8g krb5kdc[398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:01 dist-test-slave-7g8g krb5kdc[398](info): set up 2 sockets
Jun 20 23:10:01 dist-test-slave-7g8g krb5kdc[398](info): commencing operation
krb5kdc: starting...
W20260620 23:10:02.298956 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:02 dist-test-slave-7g8g krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:10:02.531051 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:02.845217 411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:02.845508 411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:02.845638 411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:02.852711 411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:02.852815 411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:02.852881 411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:02.852962 411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:02.853017 411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:02.862689 411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:02.863976 411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:02.865238 411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:02.876538 416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:02.876530 417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:02.876567 419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:02.877297 411 server_base.cc:1034] running on GCE node
I20260620 23:10:02.878062 411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:02.879513 411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:02.880712 411 hybrid_clock.cc:648] HybridClock initialized: now 1781997002880661 us; error 46 us; skew 500 ppm
I20260620 23:10:02.881067 411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:02 dist-test-slave-7g8g krb5kdc[398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:02.884079 411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:10:02.885483 411 webserver.cc:460] Webserver started at http://127.0.0.1:43657/ using document root <none> and password file <none>
I20260620 23:10:02.886145 411 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:02.886260 411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:02.886534 411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:02.889441 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "9703d0242ec64bc9af431a9417c16a72"
format_stamp: "Formatted at 2026-06-20 23:10:02 on dist-test-slave-7g8g"
server_key: "080e5400199c140e8c93e7347ff59e58"
server_key_iv: "f67fd15ef2f53a76c828f405dc084271"
server_key_version: "encryptionkey@0"
I20260620 23:10:02.890170 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "9703d0242ec64bc9af431a9417c16a72"
format_stamp: "Formatted at 2026-06-20 23:10:02 on dist-test-slave-7g8g"
server_key: "080e5400199c140e8c93e7347ff59e58"
server_key_iv: "f67fd15ef2f53a76c828f405dc084271"
server_key_version: "encryptionkey@0"
I20260620 23:10:02.895223 411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 23:10:02.898566 426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:02.899879 411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:02.900041 411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "9703d0242ec64bc9af431a9417c16a72"
format_stamp: "Formatted at 2026-06-20 23:10:02 on dist-test-slave-7g8g"
server_key: "080e5400199c140e8c93e7347ff59e58"
server_key_iv: "f67fd15ef2f53a76c828f405dc084271"
server_key_version: "encryptionkey@0"
I20260620 23:10:02.900223 411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:02.905649 411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:02.909458 411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:02.909718 411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:02.926149 411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36775
I20260620 23:10:02.926183 477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36775 every 8 connection(s)
I20260620 23:10:02.927570 411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:02.931303 478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:02.933619 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 411
I20260620 23:10:02.933833 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:02.934450 12297 external_mini_cluster.cc:1428] Setting key 22247e2a33b63e24a6b9cd1e55dfb472
I20260620 23:10:02.942790 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:02.936519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45197 (local address 127.0.0.1:36775)
0620 23:10:02.937378 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:02.937395 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:02.937452 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:10:02.939446 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:02.939470 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:02.939856 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:02.940223 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:02.941250 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:02.941258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:02.941272 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:02.941286 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:02.941342 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:02.941361 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:02.941558 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:02.941682 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 23:10:02.941864 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":389,"threads_started":1}
I20260620 23:10:02.946425 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72: Bootstrap starting.
I20260620 23:10:02.946702 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:02.949847 478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:02.951099 478 log.cc:824] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:02.953361 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:02.950995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32879 (local address 127.0.0.1:36775)
0620 23:10:02.951169 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:02.951174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:02.951457 (+ 283us) server_negotiation.cc:407] Connection header received
0620 23:10:02.951637 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:02.951643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:02.951710 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:02.951828 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:02.952780 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:02.952784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:02.952788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:02.952802 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:02.952841 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:02.952856 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:02.952946 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:02.953066 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 23:10:02.953111 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:10:02.954385 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72: No bootstrap required, opened a new log
I20260620 23:10:02.961604 478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d0242ec64bc9af431a9417c16a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 23:10:02.961949 478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:02.962028 478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9703d0242ec64bc9af431a9417c16a72, State: Initialized, Role: FOLLOWER
I20260620 23:10:02.962548 478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d0242ec64bc9af431a9417c16a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 23:10:02.962744 478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:02.962837 478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:02.962990 478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:02.965034 478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d0242ec64bc9af431a9417c16a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 23:10:02.965528 478 leader_election.cc:302] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9703d0242ec64bc9af431a9417c16a72; no voters:
I20260620 23:10:02.966154 478 leader_election.cc:288] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:02.966265 488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:02.966842 488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [term 1 LEADER]: Becoming Leader. State: Replica: 9703d0242ec64bc9af431a9417c16a72, State: Running, Role: LEADER
I20260620 23:10:02.967375 488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d0242ec64bc9af431a9417c16a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 23:10:02.968070 478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:02.969998 490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9703d0242ec64bc9af431a9417c16a72. Latest consensus state: current_term: 1 leader_uuid: "9703d0242ec64bc9af431a9417c16a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d0242ec64bc9af431a9417c16a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260620 23:10:02.970326 490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:02.975039 489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9703d0242ec64bc9af431a9417c16a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9703d0242ec64bc9af431a9417c16a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260620 23:10:02.975445 489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:02.976740 495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:02.982373 495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:02.990589 495 catalog_manager.cc:1347] Generated new cluster ID: 958fe3626a71461da78dd67becb39de0
I20260620 23:10:02.990705 495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:03.013585 495 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:03.014003 495 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:03.023818 495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9703d0242ec64bc9af431a9417c16a72: Generated new TSK 0
I20260620 23:10:03.024621 495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:03.039776 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 411
W20260620 23:10:03.040882 473 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45197
W20260620 23:10:03.041517 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 411: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:03.041666 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 411
2026-06-20T23:10:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1744 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:03 dist-test-slave-7g8g krb5kdc[509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:03 dist-test-slave-7g8g krb5kdc[509](info): set up 2 sockets
Jun 20 23:10:03 dist-test-slave-7g8g krb5kdc[509](info): commencing operation
krb5kdc: starting...
W20260620 23:10:04.042906 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:04 dist-test-slave-7g8g krb5kdc[509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:10:04.279976 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:04.592656 522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:04.592932 522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:04.593030 522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:04.600055 522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:04.600162 522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:04.600214 522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:04.600273 522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:04.600315 522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:04.609848 522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:04.611053 522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:04.612226 522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:04.622944 527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:04.623008 528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:04.623029 530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:04.623792 522 server_base.cc:1034] running on GCE node
I20260620 23:10:04.624501 522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:04.625749 522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:04.626963 522 hybrid_clock.cc:648] HybridClock initialized: now 1781997004626900 us; error 30 us; skew 500 ppm
I20260620 23:10:04.627269 522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:04 dist-test-slave-7g8g krb5kdc[509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:04.634292 522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:10:04.635723 522 webserver.cc:460] Webserver started at http://127.0.0.1:33255/ using document root <none> and password file <none>
I20260620 23:10:04.636343 522 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:04.636471 522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:04.636739 522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:04.639611 522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "a8e500859b544f579764949dba29a783"
format_stamp: "Formatted at 2026-06-20 23:10:04 on dist-test-slave-7g8g"
server_key: "6b2fe9a54d49d3def9b9d43a856c139a"
server_key_iv: "dbb511b980718374166f6c7a3249f4b3"
server_key_version: "encryptionkey@0"
I20260620 23:10:04.640331 522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "a8e500859b544f579764949dba29a783"
format_stamp: "Formatted at 2026-06-20 23:10:04 on dist-test-slave-7g8g"
server_key: "6b2fe9a54d49d3def9b9d43a856c139a"
server_key_iv: "dbb511b980718374166f6c7a3249f4b3"
server_key_version: "encryptionkey@0"
I20260620 23:10:04.644975 522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:10:04.648032 537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:04.649192 522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:10:04.649341 522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "a8e500859b544f579764949dba29a783"
format_stamp: "Formatted at 2026-06-20 23:10:04 on dist-test-slave-7g8g"
server_key: "6b2fe9a54d49d3def9b9d43a856c139a"
server_key_iv: "dbb511b980718374166f6c7a3249f4b3"
server_key_version: "encryptionkey@0"
I20260620 23:10:04.649545 522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:04.664074 522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:04.668766 522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:04.669082 522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:04.686012 522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46369
I20260620 23:10:04.686035 588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46369 every 8 connection(s)
I20260620 23:10:04.687289 522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:04.690199 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 522
I20260620 23:10:04.690361 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:04.690677 589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:04.690778 12297 external_mini_cluster.cc:1428] Setting key 4105c38f6763f9f4d393fe10af4639b0
I20260620 23:10:04.703934 589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783: Bootstrap starting.
Jun 20 23:10:04 dist-test-slave-7g8g krb5kdc[509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:10:04.706801 589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:04.707798 589 log.cc:824] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:04.710657 589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783: No bootstrap required, opened a new log
I20260620 23:10:04.712282 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:04.692444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43377 (local address 127.0.0.1:46369)
0620 23:10:04.693214 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:04.693225 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:04.693269 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:10:04.694954 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:04.694983 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:04.695378 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:04.695675 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:04.697286 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:04.698535 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:04.700358 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:04.701020 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:04.701147 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:04.705565 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:04.705573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:04.705585 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:04.705618 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:04.708567 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:04.709270 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:04.709276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:04.709282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:04.709448 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:04.709840 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:04.709843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:04.709846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:04.710309 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:04.710527 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:04.710997 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 23:10:04.711327 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":330,"threads_started":1}
I20260620 23:10:04.715612 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:04.719298 589 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e500859b544f579764949dba29a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46369 } }
I20260620 23:10:04.719734 589 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:04.719832 589 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8e500859b544f579764949dba29a783, State: Initialized, Role: FOLLOWER
I20260620 23:10:04.720407 589 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e500859b544f579764949dba29a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46369 } }
I20260620 23:10:04.720676 589 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:04.720810 589 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:04.721015 589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:04.723881 589 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e500859b544f579764949dba29a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46369 } }
I20260620 23:10:04.724417 589 leader_election.cc:302] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8e500859b544f579764949dba29a783; no voters:
I20260620 23:10:04.725009 589 leader_election.cc:288] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:04.725087 599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:04.727094 589 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:04.727243 599 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [term 1 LEADER]: Becoming Leader. State: Replica: a8e500859b544f579764949dba29a783, State: Running, Role: LEADER
I20260620 23:10:04.727864 599 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e500859b544f579764949dba29a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46369 } }
I20260620 23:10:04.731151 601 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8e500859b544f579764949dba29a783. Latest consensus state: current_term: 1 leader_uuid: "a8e500859b544f579764949dba29a783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e500859b544f579764949dba29a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46369 } } }
I20260620 23:10:04.731527 601 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:04.734176 600 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8e500859b544f579764949dba29a783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e500859b544f579764949dba29a783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46369 } } }
I20260620 23:10:04.735019 600 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:04.737407 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:04.720710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43055 (local address 127.0.0.1:46369)
0620 23:10:04.720902 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:04.720908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:04.721287 (+ 379us) server_negotiation.cc:407] Connection header received
0620 23:10:04.721458 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:04.721462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:04.721517 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:04.721589 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:04.723285 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:04.723952 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:04.725635 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:04.726259 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:04.726350 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:04.729468 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:04.729476 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:04.729482 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:04.729521 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:04.733312 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:04.734174 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:04.734180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:04.734184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:04.734344 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:04.734816 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:04.734821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:04.734825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:04.735295 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:04.735560 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:04.736803 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0620 23:10:04.737030 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:10:04.740713 609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:04.745373 609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:04.754007 609 catalog_manager.cc:1347] Generated new cluster ID: 6b8429c9f460465a817d4f2eb059abe5
I20260620 23:10:04.754109 609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:04.786744 609 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:04.788048 609 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:04.796357 609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8e500859b544f579764949dba29a783: Generated new TSK 0
I20260620 23:10:04.797183 609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:04.804987 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 522
I20260620 23:10:04.805990 584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:04.899492 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 522
2026-06-20T23:10:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:04 dist-test-slave-7g8g krb5kdc[622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:04 dist-test-slave-7g8g krb5kdc[622](info): set up 2 sockets
Jun 20 23:10:04 dist-test-slave-7g8g krb5kdc[622](info): commencing operation
krb5kdc: starting...
W20260620 23:10:05.906895 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:06 dist-test-slave-7g8g krb5kdc[622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:10:06.140480 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:06.450613 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:06.450922 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:06.451079 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:06.458065 635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:06.458181 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:06.458261 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:06.458343 635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:06.458406 635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:06.468077 635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:06.469345 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:06.470784 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:06.481729 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:06.481781 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:06.481822 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:06.482607 635 server_base.cc:1034] running on GCE node
I20260620 23:10:06.483384 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:06.484684 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:06.485877 635 hybrid_clock.cc:648] HybridClock initialized: now 1781997006485831 us; error 39 us; skew 500 ppm
I20260620 23:10:06.486189 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:06 dist-test-slave-7g8g krb5kdc[622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:06.489187 635 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:10:06.490587 635 webserver.cc:460] Webserver started at http://127.0.0.1:35867/ using document root <none> and password file <none>
I20260620 23:10:06.491284 635 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:06.491398 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:06.491669 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:06.494447 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "bb2389cf5da24b96b55f784dd6408610"
format_stamp: "Formatted at 2026-06-20 23:10:06 on dist-test-slave-7g8g"
server_key: "a17c15ccab00b2eb45df91520b89ca96"
server_key_iv: "086cc825563059744d7c00296011c9ff"
server_key_version: "encryptionkey@0"
I20260620 23:10:06.495220 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "bb2389cf5da24b96b55f784dd6408610"
format_stamp: "Formatted at 2026-06-20 23:10:06 on dist-test-slave-7g8g"
server_key: "a17c15ccab00b2eb45df91520b89ca96"
server_key_iv: "086cc825563059744d7c00296011c9ff"
server_key_version: "encryptionkey@0"
I20260620 23:10:06.500000 635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:10:06.503151 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:06.504335 635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:10:06.504484 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "bb2389cf5da24b96b55f784dd6408610"
format_stamp: "Formatted at 2026-06-20 23:10:06 on dist-test-slave-7g8g"
server_key: "a17c15ccab00b2eb45df91520b89ca96"
server_key_iv: "086cc825563059744d7c00296011c9ff"
server_key_version: "encryptionkey@0"
I20260620 23:10:06.504650 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:06.520115 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:06.524099 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:06.524350 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:06.541267 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40083
I20260620 23:10:06.541290 701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40083 every 8 connection(s)
I20260620 23:10:06.542644 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:06.546241 702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:06.551975 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 635
I20260620 23:10:06.552134 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:06.552525 12297 external_mini_cluster.cc:1428] Setting key 8b563fe6812a98c16ff5bb7821a3e0bc
I20260620 23:10:06.556574 702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610: Bootstrap starting.
I20260620 23:10:06.559930 702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:06.561190 702 log.cc:824] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:06.564360 702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610: No bootstrap required, opened a new log
I20260620 23:10:06.565807 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:06.554269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54769 (local address 127.0.0.1:40083)
0620 23:10:06.555070 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:06.555081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:06.555122 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:10:06.556821 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:06.556846 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:06.557242 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:06.557579 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:06.558185 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:06.559532 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:06.561368 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:06.562365 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:06.562549 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:06.563646 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:06.563654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:06.563669 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:06.564181 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:06.564250 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:06.564273 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:06.564370 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:06.564609 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 23:10:06.564867 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":342,"threads_started":1}
I20260620 23:10:06.569303 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:06.571991 702 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2389cf5da24b96b55f784dd6408610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40083 } }
I20260620 23:10:06.572346 702 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:06.572494 702 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2389cf5da24b96b55f784dd6408610, State: Initialized, Role: FOLLOWER
I20260620 23:10:06.573024 702 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2389cf5da24b96b55f784dd6408610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40083 } }
I20260620 23:10:06.573243 702 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:06.573414 702 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:06.573632 702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:06.576472 702 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2389cf5da24b96b55f784dd6408610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40083 } }
I20260620 23:10:06.577055 702 leader_election.cc:302] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb2389cf5da24b96b55f784dd6408610; no voters:
I20260620 23:10:06.577705 702 leader_election.cc:288] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:06.577842 712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:06.579183 712 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [term 1 LEADER]: Becoming Leader. State: Replica: bb2389cf5da24b96b55f784dd6408610, State: Running, Role: LEADER
I20260620 23:10:06.579766 712 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2389cf5da24b96b55f784dd6408610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40083 } }
I20260620 23:10:06.580206 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:06.573419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34025 (local address 127.0.0.1:40083)
0620 23:10:06.573605 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:06.573612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:06.574038 (+ 426us) server_negotiation.cc:407] Connection header received
0620 23:10:06.574229 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:06.574233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:06.574299 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:06.574386 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:06.574963 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:06.575675 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:06.577434 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:06.578122 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:06.578252 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:06.579439 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:06.579445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:06.579450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:06.579476 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:06.579520 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:06.579539 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:06.579636 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:06.579794 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 23:10:06.579881 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:10:06.580350 702 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:06.582417 714 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb2389cf5da24b96b55f784dd6408610. Latest consensus state: current_term: 1 leader_uuid: "bb2389cf5da24b96b55f784dd6408610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2389cf5da24b96b55f784dd6408610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40083 } } }
I20260620 23:10:06.582767 714 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:06.586144 717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:06.586043 713 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb2389cf5da24b96b55f784dd6408610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb2389cf5da24b96b55f784dd6408610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40083 } } }
I20260620 23:10:06.586346 713 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:06.591084 717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:06.600169 717 catalog_manager.cc:1347] Generated new cluster ID: ffc017acc9614ab8ba18170454a80061
I20260620 23:10:06.600289 717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:06.621163 717 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:06.622643 717 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:06.628624 717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb2389cf5da24b96b55f784dd6408610: Generated new TSK 0
I20260620 23:10:06.629611 717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:06.638578 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 635
W20260620 23:10:06.639859 697 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54769
W20260620 23:10:06.640529 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 635: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:06.640653 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 635
2026-06-20T23:10:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1740 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:06 dist-test-slave-7g8g krb5kdc[733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:06 dist-test-slave-7g8g krb5kdc[733](info): set up 2 sockets
Jun 20 23:10:06 dist-test-slave-7g8g krb5kdc[733](info): commencing operation
krb5kdc: starting...
W20260620 23:10:07.650998 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:07 dist-test-slave-7g8g krb5kdc[733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:10:07.887547 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38543
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:08.205206 746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:08.205485 746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:08.205575 746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:08.212661 746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:08.212779 746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:08.212826 746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:08.212884 746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:08.212926 746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:08.222631 746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:08.223918 746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:08.225227 746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:08.236487 752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:08.236562 754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:08.236506 751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:08.237207 746 server_base.cc:1034] running on GCE node
I20260620 23:10:08.238107 746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:08.239540 746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:08.240741 746 hybrid_clock.cc:648] HybridClock initialized: now 1781997008240695 us; error 36 us; skew 500 ppm
I20260620 23:10:08.241065 746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:08 dist-test-slave-7g8g krb5kdc[733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:08.243957 746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:10:08.245328 746 webserver.cc:460] Webserver started at http://127.0.0.1:36049/ using document root <none> and password file <none>
I20260620 23:10:08.245968 746 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:08.246079 746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:08.246351 746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:08.249205 746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "8faae5b2199f410eb58b2b21e33e84ef"
format_stamp: "Formatted at 2026-06-20 23:10:08 on dist-test-slave-7g8g"
server_key: "6f5e9572c2115de57a422dc1e5c78189"
server_key_iv: "52b1170c682d7ef463a47eb34acb0f6e"
server_key_version: "encryptionkey@0"
I20260620 23:10:08.249958 746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "8faae5b2199f410eb58b2b21e33e84ef"
format_stamp: "Formatted at 2026-06-20 23:10:08 on dist-test-slave-7g8g"
server_key: "6f5e9572c2115de57a422dc1e5c78189"
server_key_iv: "52b1170c682d7ef463a47eb34acb0f6e"
server_key_version: "encryptionkey@0"
I20260620 23:10:08.254724 746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 23:10:08.257947 761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:08.259147 746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:10:08.259292 746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "8faae5b2199f410eb58b2b21e33e84ef"
format_stamp: "Formatted at 2026-06-20 23:10:08 on dist-test-slave-7g8g"
server_key: "6f5e9572c2115de57a422dc1e5c78189"
server_key_iv: "52b1170c682d7ef463a47eb34acb0f6e"
server_key_version: "encryptionkey@0"
I20260620 23:10:08.259455 746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:08.264626 746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:08.268173 746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:08.268416 746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:08.284536 746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38543
I20260620 23:10:08.284571 812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38543 every 8 connection(s)
I20260620 23:10:08.285828 746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:08.289364 813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:08.290756 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 746
I20260620 23:10:08.290911 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:08.291365 12297 external_mini_cluster.cc:1428] Setting key 4574bf58e83b77cf506807ebcfedaba3
I20260620 23:10:08.298611 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:08.293063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47505 (local address 127.0.0.1:38543)
0620 23:10:08.293862 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:08.293875 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:08.293920 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:10:08.295539 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:08.295559 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:08.295908 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:08.296205 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:08.297203 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:08.297208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:08.297220 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:08.297233 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:08.297290 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:08.297308 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:08.297471 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:08.297571 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 23:10:08.297732 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":338,"threads_started":1}
I20260620 23:10:08.302134 813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef: Bootstrap starting.
I20260620 23:10:08.302235 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:08.304999 813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:08.305938 813 log.cc:824] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:08.308598 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:08.306308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33421 (local address 127.0.0.1:38543)
0620 23:10:08.306485 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:08.306491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:08.306988 (+ 497us) server_negotiation.cc:407] Connection header received
0620 23:10:08.307123 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:08.307126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:08.307170 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:08.307242 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:08.308068 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:08.308072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:08.308075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:08.308088 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:08.308122 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:08.308135 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:08.308213 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:08.308330 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 23:10:08.308371 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:10:08.308887 813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef: No bootstrap required, opened a new log
I20260620 23:10:08.315889 813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faae5b2199f410eb58b2b21e33e84ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } }
I20260620 23:10:08.316200 813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:08.316267 813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8faae5b2199f410eb58b2b21e33e84ef, State: Initialized, Role: FOLLOWER
I20260620 23:10:08.316761 813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faae5b2199f410eb58b2b21e33e84ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } }
I20260620 23:10:08.316953 813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:08.317045 813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:08.317157 813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:08.319134 813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faae5b2199f410eb58b2b21e33e84ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } }
I20260620 23:10:08.319664 813 leader_election.cc:302] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8faae5b2199f410eb58b2b21e33e84ef; no voters:
I20260620 23:10:08.320134 813 leader_election.cc:288] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:08.320247 823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:08.320881 823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [term 1 LEADER]: Becoming Leader. State: Replica: 8faae5b2199f410eb58b2b21e33e84ef, State: Running, Role: LEADER
I20260620 23:10:08.321352 823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faae5b2199f410eb58b2b21e33e84ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } }
I20260620 23:10:08.322005 813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:08.323613 825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8faae5b2199f410eb58b2b21e33e84ef. Latest consensus state: current_term: 1 leader_uuid: "8faae5b2199f410eb58b2b21e33e84ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faae5b2199f410eb58b2b21e33e84ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } } }
I20260620 23:10:08.324066 825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:08.323768 824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8faae5b2199f410eb58b2b21e33e84ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8faae5b2199f410eb58b2b21e33e84ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38543 } } }
I20260620 23:10:08.324479 824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:08.329818 830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:08.334625 830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:08.342586 830 catalog_manager.cc:1347] Generated new cluster ID: 0b0ec5d632d3486e9f7efbb0d60e470e
I20260620 23:10:08.342679 830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:08.369804 830 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:08.370198 830 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:08.378371 830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8faae5b2199f410eb58b2b21e33e84ef: Generated new TSK 0
I20260620 23:10:08.379302 830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:08.398114 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 746
W20260620 23:10:08.399224 808 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47505
W20260620 23:10:08.399909 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 746: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:08.400036 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 746
2026-06-20T23:10:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1760 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:08 dist-test-slave-7g8g krb5kdc[844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:08 dist-test-slave-7g8g krb5kdc[844](info): set up 2 sockets
Jun 20 23:10:08 dist-test-slave-7g8g krb5kdc[844](info): commencing operation
krb5kdc: starting...
W20260620 23:10:09.418851 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:09 dist-test-slave-7g8g krb5kdc[844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:10:09.643419 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:09.954612 857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:09.954893 857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:09.955039 857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:09.962132 857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:09.962241 857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:09.962317 857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:09.962399 857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:09.962463 857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:09.972122 857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:09.973338 857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:09.974566 857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:09.985206 863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:09.985242 865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:09.985455 862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:09.986177 857 server_base.cc:1034] running on GCE node
I20260620 23:10:09.986865 857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:09.988168 857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:09.989355 857 hybrid_clock.cc:648] HybridClock initialized: now 1781997009989309 us; error 42 us; skew 500 ppm
I20260620 23:10:09.989660 857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:09 dist-test-slave-7g8g krb5kdc[844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:09.996896 857 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:10:09.998332 857 webserver.cc:460] Webserver started at http://127.0.0.1:40095/ using document root <none> and password file <none>
I20260620 23:10:09.999001 857 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:09.999116 857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:09.999398 857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:10.002254 857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "b0f36a4338894c29be648a55663aa563"
format_stamp: "Formatted at 2026-06-20 23:10:09 on dist-test-slave-7g8g"
server_key: "f5d7013a9e73d9900ed333877b9d2f2c"
server_key_iv: "e1aaa0958c6a7d5031870c45f4714a2f"
server_key_version: "encryptionkey@0"
I20260620 23:10:10.003027 857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "b0f36a4338894c29be648a55663aa563"
format_stamp: "Formatted at 2026-06-20 23:10:09 on dist-test-slave-7g8g"
server_key: "f5d7013a9e73d9900ed333877b9d2f2c"
server_key_iv: "e1aaa0958c6a7d5031870c45f4714a2f"
server_key_version: "encryptionkey@0"
I20260620 23:10:10.007784 857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:10:10.010948 872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:10.012118 857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 23:10:10.012262 857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "b0f36a4338894c29be648a55663aa563"
format_stamp: "Formatted at 2026-06-20 23:10:09 on dist-test-slave-7g8g"
server_key: "f5d7013a9e73d9900ed333877b9d2f2c"
server_key_iv: "e1aaa0958c6a7d5031870c45f4714a2f"
server_key_version: "encryptionkey@0"
I20260620 23:10:10.012426 857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:10.028003 857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:10.031551 857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:10.031800 857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:10.049433 857 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40135
I20260620 23:10:10.049435 923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40135 every 8 connection(s)
I20260620 23:10:10.050786 857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:10.053637 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 857
I20260620 23:10:10.053803 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:10.054315 12297 external_mini_cluster.cc:1428] Setting key dffd2b10b459f3ba24f919ad51b70506
I20260620 23:10:10.054510 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:10.067754 924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563: Bootstrap starting.
Jun 20 23:10:10 dist-test-slave-7g8g krb5kdc[844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:10:10.071139 924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:10.072412 924 log.cc:824] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:10.075522 924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563: No bootstrap required, opened a new log
I20260620 23:10:10.077277 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:10.056034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:40435 (local address 127.0.0.1:40135)
0620 23:10:10.056731 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:10.056742 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:10.056786 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:10:10.058815 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:10.058836 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:10.059278 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:10.059592 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:10.061255 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:10.062459 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:10.064112 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:10.064776 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:10.064923 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:10.069299 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:10.069309 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:10.069336 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:10.069386 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:10.072988 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:10.073855 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:10.073863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:10.073870 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:10.074051 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:10.074435 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:10.074439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:10.074442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:10.075097 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:10.075405 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:10.075729 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 23:10:10.076050 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":324,"threads_started":1}
I20260620 23:10:10.080832 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:10.083436 924 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f36a4338894c29be648a55663aa563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40135 } }
I20260620 23:10:10.083781 924 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:10.083880 924 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f36a4338894c29be648a55663aa563, State: Initialized, Role: FOLLOWER
I20260620 23:10:10.084417 924 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f36a4338894c29be648a55663aa563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40135 } }
I20260620 23:10:10.084635 924 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:10.084761 924 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:10.084964 924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:10.087786 924 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f36a4338894c29be648a55663aa563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40135 } }
I20260620 23:10:10.088390 924 leader_election.cc:302] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0f36a4338894c29be648a55663aa563; no voters:
I20260620 23:10:10.089031 924 leader_election.cc:288] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:10.089126 934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:10.091691 934 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [term 1 LEADER]: Becoming Leader. State: Replica: b0f36a4338894c29be648a55663aa563, State: Running, Role: LEADER
I20260620 23:10:10.092252 934 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f36a4338894c29be648a55663aa563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40135 } }
I20260620 23:10:10.092938 924 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:10.094924 936 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0f36a4338894c29be648a55663aa563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f36a4338894c29be648a55663aa563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40135 } } }
I20260620 23:10:10.095422 935 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0f36a4338894c29be648a55663aa563. Latest consensus state: current_term: 1 leader_uuid: "b0f36a4338894c29be648a55663aa563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f36a4338894c29be648a55663aa563" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40135 } } }
I20260620 23:10:10.095726 935 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:10.097041 936 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:10.099493 939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:10.100258 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:10.085006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:59911 (local address 127.0.0.1:40135)
0620 23:10:10.085159 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:10.085164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:10.085570 (+ 406us) server_negotiation.cc:407] Connection header received
0620 23:10:10.085686 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:10.085691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:10.085770 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:10.085873 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:10.087470 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:10.088397 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:10.090110 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:10.090530 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:10.090637 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:10.093287 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:10.093292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:10.093296 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:10.093329 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:10.096822 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:10.097616 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:10.097623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:10.097627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:10.097783 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:10.098293 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:10.098298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:10.098301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:10.098582 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:10.098838 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:10.099760 (+ 922us) server_negotiation.cc:299] Negotiation successful
0620 23:10:10.099928 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:10:10.104912 939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:10.117403 939 catalog_manager.cc:1347] Generated new cluster ID: 201abed2c6c4474ca6750efb9ed996bb
I20260620 23:10:10.117508 939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:10.138504 939 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:10.140332 939 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:10.150981 939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0f36a4338894c29be648a55663aa563: Generated new TSK 0
I20260620 23:10:10.152004 939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:10.168009 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 857
I20260620 23:10:10.169754 919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:10.260783 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 857
W20260620 23:10:10.274240 842 connection.cc:535] client connection to 127.0.0.1:40135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40135): Connection reset by peer (error 104)
2026-06-20T23:10:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1860 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:10 dist-test-slave-7g8g krb5kdc[956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:10 dist-test-slave-7g8g krb5kdc[956](info): set up 2 sockets
Jun 20 23:10:10 dist-test-slave-7g8g krb5kdc[956](info): commencing operation
krb5kdc: starting...
W20260620 23:10:11.270969 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:11 dist-test-slave-7g8g krb5kdc[956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:10:11.497959 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:11.808967 969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:11.809273 969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:11.809422 969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:11.816712 969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:11.816844 969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:11.816924 969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:11.817003 969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:11.817061 969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:11.826864 969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:11.828238 969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:11.829605 969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:11.840482 975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:11.840482 977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:11.840591 974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:11.841217 969 server_base.cc:1034] running on GCE node
I20260620 23:10:11.841866 969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:11.843138 969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:11.844338 969 hybrid_clock.cc:648] HybridClock initialized: now 1781997011844301 us; error 28 us; skew 500 ppm
I20260620 23:10:11.844653 969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:11 dist-test-slave-7g8g krb5kdc[956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:11.847590 969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:10:11.848996 969 webserver.cc:460] Webserver started at http://127.0.0.1:45089/ using document root <none> and password file <none>
I20260620 23:10:11.849597 969 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:11.849691 969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:11.849936 969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:11.852823 969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "f503454446b0467480d545ec96ea3130"
format_stamp: "Formatted at 2026-06-20 23:10:11 on dist-test-slave-7g8g"
server_key: "f00d5317185d60f0b328f0acba6ddec1"
server_key_iv: "c67f05f8c8f5178e79b0f02ec8b7fd25"
server_key_version: "encryptionkey@0"
I20260620 23:10:11.853513 969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "f503454446b0467480d545ec96ea3130"
format_stamp: "Formatted at 2026-06-20 23:10:11 on dist-test-slave-7g8g"
server_key: "f00d5317185d60f0b328f0acba6ddec1"
server_key_iv: "c67f05f8c8f5178e79b0f02ec8b7fd25"
server_key_version: "encryptionkey@0"
I20260620 23:10:11.858512 969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:10:11.862056 984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:11.863404 969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:11.863544 969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "f503454446b0467480d545ec96ea3130"
format_stamp: "Formatted at 2026-06-20 23:10:11 on dist-test-slave-7g8g"
server_key: "f00d5317185d60f0b328f0acba6ddec1"
server_key_iv: "c67f05f8c8f5178e79b0f02ec8b7fd25"
server_key_version: "encryptionkey@0"
I20260620 23:10:11.863718 969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:11.877828 969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:11.881954 969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:11.882210 969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:11.898425 969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40571
I20260620 23:10:11.898454 1035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40571 every 8 connection(s)
I20260620 23:10:11.899734 969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:11.903216 1036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:11.909461 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 969
I20260620 23:10:11.909643 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:11.910178 12297 external_mini_cluster.cc:1428] Setting key da27793d32774ada9902da869047f4eb
I20260620 23:10:11.913695 1036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130: Bootstrap starting.
I20260620 23:10:11.916997 1036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:11.917948 1036 log.cc:824] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:11.920650 1036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130: No bootstrap required, opened a new log
I20260620 23:10:11.922449 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:11.912004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:43351 (local address 127.0.0.1:40571)
0620 23:10:11.912707 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:11.912719 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:11.912766 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:10:11.914419 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:11.914440 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:11.914825 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:11.915171 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:11.915766 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:11.917011 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:11.918282 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:11.919012 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:11.919168 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:11.920323 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:11.920331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:11.920344 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:11.920888 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:11.920959 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:11.920983 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:11.921131 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:11.921379 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 23:10:11.921527 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":317,"threads_started":1}
I20260620 23:10:11.926100 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:11.928745 1036 raft_consensus.cc:348] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503454446b0467480d545ec96ea3130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } }
I20260620 23:10:11.929169 1036 raft_consensus.cc:374] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:11.929265 1036 raft_consensus.cc:729] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f503454446b0467480d545ec96ea3130, State: Initialized, Role: FOLLOWER
I20260620 23:10:11.929814 1036 consensus_queue.cc:260] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503454446b0467480d545ec96ea3130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } }
I20260620 23:10:11.930078 1036 raft_consensus.cc:388] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:11.930218 1036 raft_consensus.cc:482] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:11.930431 1036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:11.933275 1036 raft_consensus.cc:504] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503454446b0467480d545ec96ea3130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } }
I20260620 23:10:11.933740 1036 leader_election.cc:302] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f503454446b0467480d545ec96ea3130; no voters:
I20260620 23:10:11.934337 1036 leader_election.cc:288] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:11.934420 1046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:11.935570 1046 raft_consensus.cc:686] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [term 1 LEADER]: Becoming Leader. State: Replica: f503454446b0467480d545ec96ea3130, State: Running, Role: LEADER
I20260620 23:10:11.936126 1046 consensus_queue.cc:237] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503454446b0467480d545ec96ea3130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } }
I20260620 23:10:11.936560 1036 sys_catalog.cc:564] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:11.937235 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:11.930202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:38135 (local address 127.0.0.1:40571)
0620 23:10:11.930372 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:11.930378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:11.930762 (+ 384us) server_negotiation.cc:407] Connection header received
0620 23:10:11.930891 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:11.930896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:11.930997 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:11.931101 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:11.931535 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:11.932630 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:11.933932 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:11.934666 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:11.934806 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:11.936442 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:11.936449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:11.936454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:11.936474 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:11.936518 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:11.936539 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:11.936645 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:11.936880 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 23:10:11.936931 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:10:11.938791 1047 sys_catalog.cc:455] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f503454446b0467480d545ec96ea3130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503454446b0467480d545ec96ea3130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } } }
I20260620 23:10:11.939177 1047 sys_catalog.cc:458] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:11.939111 1048 sys_catalog.cc:455] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f503454446b0467480d545ec96ea3130. Latest consensus state: current_term: 1 leader_uuid: "f503454446b0467480d545ec96ea3130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503454446b0467480d545ec96ea3130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40571 } } }
I20260620 23:10:11.939407 1048 sys_catalog.cc:458] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:11.940774 1051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:11.946213 1051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:11.954820 1051 catalog_manager.cc:1347] Generated new cluster ID: 7c5a7712bd084efab2ce3e91a982cffb
I20260620 23:10:11.954914 1051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:11.974227 1051 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:11.975538 1051 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:11.986151 1051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f503454446b0467480d545ec96ea3130: Generated new TSK 0
I20260620 23:10:11.986984 1051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:11.994145 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 969
W20260620 23:10:11.995260 1031 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.196:43351
W20260620 23:10:11.995988 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 969: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:11.996140 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 969
W20260620 23:10:12.009472 954 connection.cc:535] client connection to 127.0.0.1:40571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40571): Connection reset by peer (error 104)
2026-06-20T23:10:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1735 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:12 dist-test-slave-7g8g krb5kdc[1067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:12 dist-test-slave-7g8g krb5kdc[1067](info): set up 2 sockets
Jun 20 23:10:12 dist-test-slave-7g8g krb5kdc[1067](info): commencing operation
krb5kdc: starting...
W20260620 23:10:13.007068 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:13 dist-test-slave-7g8g krb5kdc[1067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:10:13.245675 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:13.568668 1080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:13.568950 1080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:13.569089 1080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:13.576086 1080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:13.576212 1080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:13.576290 1080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:13.576387 1080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:13.576443 1080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:13.586105 1080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:13.587379 1080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:13.588707 1080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:13.599617 1088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:13.599684 1085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:13.599663 1086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:13.600526 1080 server_base.cc:1034] running on GCE node
I20260620 23:10:13.601274 1080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:13.602607 1080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:13.603801 1080 hybrid_clock.cc:648] HybridClock initialized: now 1781997013603758 us; error 30 us; skew 500 ppm
I20260620 23:10:13.604118 1080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:13 dist-test-slave-7g8g krb5kdc[1067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:13.606971 1080 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:10:13.608356 1080 webserver.cc:460] Webserver started at http://127.0.0.1:39179/ using document root <none> and password file <none>
I20260620 23:10:13.609063 1080 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:13.609179 1080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:13.609467 1080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:13.612499 1080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "e4800251865444d8a9294ad527c34e5c"
format_stamp: "Formatted at 2026-06-20 23:10:13 on dist-test-slave-7g8g"
server_key: "8698cc2cfc5894cc04d87a9d0dfbbd65"
server_key_iv: "17013c428c961620927a32e8017260ef"
server_key_version: "encryptionkey@0"
I20260620 23:10:13.613250 1080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "e4800251865444d8a9294ad527c34e5c"
format_stamp: "Formatted at 2026-06-20 23:10:13 on dist-test-slave-7g8g"
server_key: "8698cc2cfc5894cc04d87a9d0dfbbd65"
server_key_iv: "17013c428c961620927a32e8017260ef"
server_key_version: "encryptionkey@0"
I20260620 23:10:13.618265 1080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 23:10:13.621764 1095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:13.623143 1080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:10:13.623297 1080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "e4800251865444d8a9294ad527c34e5c"
format_stamp: "Formatted at 2026-06-20 23:10:13 on dist-test-slave-7g8g"
server_key: "8698cc2cfc5894cc04d87a9d0dfbbd65"
server_key_iv: "17013c428c961620927a32e8017260ef"
server_key_version: "encryptionkey@0"
I20260620 23:10:13.623476 1080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:13.629076 1080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:13.632798 1080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:13.633049 1080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:13.649503 1080 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34139
I20260620 23:10:13.649523 1146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34139 every 8 connection(s)
I20260620 23:10:13.650803 1080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:13.654289 1147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:13.659766 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 1080
I20260620 23:10:13.659930 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:13.660328 12297 external_mini_cluster.cc:1428] Setting key acb2e606d672bee62ef250b727d1974f
I20260620 23:10:13.664533 1147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c: Bootstrap starting.
I20260620 23:10:13.667781 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:13.662081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:41813 (local address 127.0.0.1:34139)
0620 23:10:13.662871 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:13.662882 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:13.662920 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:10:13.664567 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:13.664589 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:13.664944 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:13.665247 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:13.666319 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:13.666324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:13.666336 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:13.666350 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:13.666415 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:13.666434 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:13.666599 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:13.666695 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 23:10:13.666853 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":341,"threads_started":1}
I20260620 23:10:13.667902 1147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:13.668982 1147 log.cc:824] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:13.671406 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:13.671767 1147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c: No bootstrap required, opened a new log
I20260620 23:10:13.677938 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:13.675664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:42029 (local address 127.0.0.1:34139)
0620 23:10:13.675866 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:13.675871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:13.676171 (+ 300us) server_negotiation.cc:407] Connection header received
0620 23:10:13.676320 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:13.676324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:13.676383 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:13.676465 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:13.677336 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:13.677339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:13.677342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:13.677356 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:13.677395 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:13.677412 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:13.677508 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:13.677602 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:10:13.677649 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:10:13.679908 1147 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4800251865444d8a9294ad527c34e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34139 } }
I20260620 23:10:13.680260 1147 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:13.680358 1147 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4800251865444d8a9294ad527c34e5c, State: Initialized, Role: FOLLOWER
I20260620 23:10:13.680914 1147 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4800251865444d8a9294ad527c34e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34139 } }
I20260620 23:10:13.681128 1147 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:13.681252 1147 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:13.681416 1147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:13.683838 1147 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4800251865444d8a9294ad527c34e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34139 } }
I20260620 23:10:13.684314 1147 leader_election.cc:302] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4800251865444d8a9294ad527c34e5c; no voters:
I20260620 23:10:13.684851 1147 leader_election.cc:288] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:13.684978 1157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:13.685572 1157 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [term 1 LEADER]: Becoming Leader. State: Replica: e4800251865444d8a9294ad527c34e5c, State: Running, Role: LEADER
I20260620 23:10:13.686127 1157 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4800251865444d8a9294ad527c34e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34139 } }
I20260620 23:10:13.686760 1147 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:13.688845 1158 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4800251865444d8a9294ad527c34e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4800251865444d8a9294ad527c34e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34139 } } }
I20260620 23:10:13.688964 1159 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4800251865444d8a9294ad527c34e5c. Latest consensus state: current_term: 1 leader_uuid: "e4800251865444d8a9294ad527c34e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4800251865444d8a9294ad527c34e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34139 } } }
I20260620 23:10:13.689244 1158 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:13.689256 1159 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:13.691589 1165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:13.695685 1165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:13.704353 1165 catalog_manager.cc:1347] Generated new cluster ID: a240d503b3fb492591c3a84640965e98
I20260620 23:10:13.704478 1165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:13.733758 1165 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:13.734308 1165 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:13.741271 1165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4800251865444d8a9294ad527c34e5c: Generated new TSK 0
I20260620 23:10:13.742281 1165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:13.750360 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1080
W20260620 23:10:13.751462 1142 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.196:41813
W20260620 23:10:13.752117 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1080: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:13.752264 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 1080
2026-06-20T23:10:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1756 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:13 dist-test-slave-7g8g krb5kdc[1178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:13 dist-test-slave-7g8g krb5kdc[1178](info): set up 2 sockets
Jun 20 23:10:13 dist-test-slave-7g8g krb5kdc[1178](info): commencing operation
krb5kdc: starting...
W20260620 23:10:14.766911 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:14 dist-test-slave-7g8g krb5kdc[1178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:10:15.002074 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:15.322788 1191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:15.323151 1191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:15.323261 1191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:15.330313 1191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:15.330415 1191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:15.330461 1191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:15.330516 1191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:15.330559 1191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:15.340498 1191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:15.341706 1191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:15.342947 1191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:15.354231 1196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:15.354317 1199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:15.354282 1197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:15.355106 1191 server_base.cc:1034] running on GCE node
I20260620 23:10:15.355896 1191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:15.357339 1191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:15.358533 1191 hybrid_clock.cc:648] HybridClock initialized: now 1781997015358491 us; error 35 us; skew 500 ppm
I20260620 23:10:15.358872 1191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:15 dist-test-slave-7g8g krb5kdc[1178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:15.364684 1191 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:10:15.366078 1191 webserver.cc:460] Webserver started at http://127.0.0.1:35685/ using document root <none> and password file <none>
I20260620 23:10:15.366703 1191 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:15.366798 1191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:15.367087 1191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:15.369896 1191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "eb1e6a4d33724fbb959b591f3097a65c"
format_stamp: "Formatted at 2026-06-20 23:10:15 on dist-test-slave-7g8g"
server_key: "04870098b0e3038f7413b735e98085d8"
server_key_iv: "fb1f2ea15fc248a811588576a9783273"
server_key_version: "encryptionkey@0"
I20260620 23:10:15.370586 1191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "eb1e6a4d33724fbb959b591f3097a65c"
format_stamp: "Formatted at 2026-06-20 23:10:15 on dist-test-slave-7g8g"
server_key: "04870098b0e3038f7413b735e98085d8"
server_key_iv: "fb1f2ea15fc248a811588576a9783273"
server_key_version: "encryptionkey@0"
I20260620 23:10:15.375349 1191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260620 23:10:15.378459 1206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:15.379717 1191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:15.379864 1191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "eb1e6a4d33724fbb959b591f3097a65c"
format_stamp: "Formatted at 2026-06-20 23:10:15 on dist-test-slave-7g8g"
server_key: "04870098b0e3038f7413b735e98085d8"
server_key_iv: "fb1f2ea15fc248a811588576a9783273"
server_key_version: "encryptionkey@0"
I20260620 23:10:15.380023 1191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:15.402454 1191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:15.406261 1191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:15.406524 1191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:15.423009 1191 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39699
I20260620 23:10:15.423058 1257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39699 every 8 connection(s)
I20260620 23:10:15.424311 1191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:15.427773 1258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:15.433709 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 1191
I20260620 23:10:15.433866 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:15.434257 12297 external_mini_cluster.cc:1428] Setting key 2ead2ab29ac929a55e399d1fc3aaaff2
I20260620 23:10:15.438318 1258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c: Bootstrap starting.
I20260620 23:10:15.441890 1258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:15.443120 1258 log.cc:824] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:15.446126 1258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c: No bootstrap required, opened a new log
Jun 20 23:10:15 dist-test-slave-7g8g krb5kdc[1178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:10:15.455071 1258 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1e6a4d33724fbb959b591f3097a65c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39699 } }
I20260620 23:10:15.455499 1258 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:15.455610 1258 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1e6a4d33724fbb959b591f3097a65c, State: Initialized, Role: FOLLOWER
I20260620 23:10:15.456282 1258 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1e6a4d33724fbb959b591f3097a65c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39699 } }
I20260620 23:10:15.456588 1258 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:15.456743 1258 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:15.456908 1258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:15.458104 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:15.436050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:36789 (local address 127.0.0.1:39699)
0620 23:10:15.436758 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:15.436773 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:15.436820 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:10:15.439066 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:15.439092 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:15.439592 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:15.440020 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:15.441244 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:15.443208 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:15.444726 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:15.445699 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:15.445868 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:15.449718 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:15.449732 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:15.449750 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:15.449796 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:15.453643 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:15.454364 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:15.454372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:15.454381 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:15.454569 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:15.454981 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:15.454985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:15.454988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:15.455651 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:15.455944 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:15.456432 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 23:10:15.456759 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":288,"threads_started":1}
I20260620 23:10:15.459620 1258 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1e6a4d33724fbb959b591f3097a65c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39699 } }
I20260620 23:10:15.460081 1258 leader_election.cc:302] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb1e6a4d33724fbb959b591f3097a65c; no voters:
I20260620 23:10:15.460667 1258 leader_election.cc:288] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:15.460790 1263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:15.461921 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:15.461956 1263 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [term 1 LEADER]: Becoming Leader. State: Replica: eb1e6a4d33724fbb959b591f3097a65c, State: Running, Role: LEADER
I20260620 23:10:15.462491 1263 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1e6a4d33724fbb959b591f3097a65c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39699 } }
I20260620 23:10:15.462628 1258 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:15.464928 1265 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb1e6a4d33724fbb959b591f3097a65c. Latest consensus state: current_term: 1 leader_uuid: "eb1e6a4d33724fbb959b591f3097a65c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1e6a4d33724fbb959b591f3097a65c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39699 } } }
I20260620 23:10:15.465308 1265 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:15.465845 1264 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb1e6a4d33724fbb959b591f3097a65c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1e6a4d33724fbb959b591f3097a65c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39699 } } }
I20260620 23:10:15.466140 1264 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:15.474144 1270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:15.479830 1270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:15.489265 1270 catalog_manager.cc:1347] Generated new cluster ID: 316929a91fc44b9d9807daee47a3b152
I20260620 23:10:15.489374 1270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:15.498651 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:15.477661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:39601 (local address 127.0.0.1:39699)
0620 23:10:15.477859 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:15.477865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:15.485124 (+ 7259us) server_negotiation.cc:407] Connection header received
0620 23:10:15.485786 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:15.485792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:15.485856 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:15.485943 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:15.487708 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:15.488917 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:15.490316 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:15.490757 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:15.490962 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:15.493205 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:15.493210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:15.493214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:15.493245 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:15.495923 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:15.496639 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:15.496643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:15.496648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:15.496779 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:15.497210 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:15.497214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:15.497216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:15.497488 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:15.497732 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:15.498100 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 23:10:15.498285 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":70}
I20260620 23:10:15.501828 1270 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:15.503417 1270 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:15.511200 1270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb1e6a4d33724fbb959b591f3097a65c: Generated new TSK 0
I20260620 23:10:15.511978 1270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:15.519014 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1191
I20260620 23:10:15.520200 1253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:15.611460 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 1191
W20260620 23:10:15.624817 1176 connection.cc:535] client connection to 127.0.0.1:39699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39699): Connection reset by peer (error 104)
2026-06-20T23:10:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1858 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:15 dist-test-slave-7g8g krb5kdc[1291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:15 dist-test-slave-7g8g krb5kdc[1291](info): set up 2 sockets
Jun 20 23:10:15 dist-test-slave-7g8g krb5kdc[1291](info): commencing operation
krb5kdc: starting...
W20260620 23:10:16.619004 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:16 dist-test-slave-7g8g krb5kdc[1291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:10:16.858278 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:17.172708 1304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:17.172981 1304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:17.173112 1304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:17.180284 1304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:17.180390 1304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:17.180461 1304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:17.180539 1304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:17.180595 1304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:17.190307 1304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:17.191639 1304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:17.192946 1304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:17.204057 1312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:17.204164 1309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:17.204165 1310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:17.204705 1304 server_base.cc:1034] running on GCE node
I20260620 23:10:17.205412 1304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:17.206748 1304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:17.207962 1304 hybrid_clock.cc:648] HybridClock initialized: now 1781997017207902 us; error 47 us; skew 500 ppm
I20260620 23:10:17.208285 1304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:17 dist-test-slave-7g8g krb5kdc[1291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:17.211227 1304 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:10:17.212541 1304 webserver.cc:460] Webserver started at http://127.0.0.1:37237/ using document root <none> and password file <none>
I20260620 23:10:17.213207 1304 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:17.213323 1304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:17.213594 1304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:17.216516 1304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "f1eca4053e804dd7a5cea4404352bb56"
format_stamp: "Formatted at 2026-06-20 23:10:17 on dist-test-slave-7g8g"
server_key: "b9af2156221e8db23910d86e4378c439"
server_key_iv: "97623ff858dc8e9bed4ade75f1e13c4e"
server_key_version: "encryptionkey@0"
I20260620 23:10:17.217239 1304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "f1eca4053e804dd7a5cea4404352bb56"
format_stamp: "Formatted at 2026-06-20 23:10:17 on dist-test-slave-7g8g"
server_key: "b9af2156221e8db23910d86e4378c439"
server_key_iv: "97623ff858dc8e9bed4ade75f1e13c4e"
server_key_version: "encryptionkey@0"
I20260620 23:10:17.222159 1304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:10:17.225747 1319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:17.227190 1304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:17.227366 1304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "f1eca4053e804dd7a5cea4404352bb56"
format_stamp: "Formatted at 2026-06-20 23:10:17 on dist-test-slave-7g8g"
server_key: "b9af2156221e8db23910d86e4378c439"
server_key_iv: "97623ff858dc8e9bed4ade75f1e13c4e"
server_key_version: "encryptionkey@0"
I20260620 23:10:17.227620 1304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:17.250761 1304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:17.254521 1304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:17.254789 1304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:17.271971 1304 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36419
I20260620 23:10:17.271997 1370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36419 every 8 connection(s)
I20260620 23:10:17.273388 1304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:17.277098 1371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:17.279994 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 1304
I20260620 23:10:17.280164 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:17.280682 12297 external_mini_cluster.cc:1428] Setting key 93850b7c0834a798133af2446952ee13
I20260620 23:10:17.288733 1371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56: Bootstrap starting.
I20260620 23:10:17.292172 1371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:17.293269 1371 log.cc:824] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:17.293713 1374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:17.282469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:50185 (local address 127.0.0.1:36419)
0620 23:10:17.283228 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:17.283239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:17.283278 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:10:17.284886 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:17.284909 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:17.285327 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:17.285647 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:17.286244 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:17.287566 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:17.289315 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:17.289969 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:17.290097 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:17.291430 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:17.291437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:17.291451 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:17.291966 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:17.292040 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:17.292065 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:17.292200 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:17.292499 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 23:10:17.292700 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":330,"threads_started":1}
I20260620 23:10:17.296370 1371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56: No bootstrap required, opened a new log
I20260620 23:10:17.297886 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:17.306137 1371 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1eca4053e804dd7a5cea4404352bb56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260620 23:10:17.306602 1371 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:17.306703 1371 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1eca4053e804dd7a5cea4404352bb56, State: Initialized, Role: FOLLOWER
I20260620 23:10:17.307369 1371 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1eca4053e804dd7a5cea4404352bb56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260620 23:10:17.307659 1371 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:17.307798 1371 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:17.307966 1371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:17.309265 1374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:17.302912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:36671 (local address 127.0.0.1:36419)
0620 23:10:17.303094 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:17.303098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:17.303440 (+ 342us) server_negotiation.cc:407] Connection header received
0620 23:10:17.303614 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:17.303617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:17.303703 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:17.303785 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:17.304338 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:17.305106 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:17.306797 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:17.307305 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:17.307400 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:17.308549 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:17.308554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:17.308558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:17.308574 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:17.308612 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:17.308627 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:17.308741 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:17.308929 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 23:10:17.308980 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:10:17.310724 1371 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1eca4053e804dd7a5cea4404352bb56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260620 23:10:17.311285 1371 leader_election.cc:302] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1eca4053e804dd7a5cea4404352bb56; no voters:
I20260620 23:10:17.311797 1371 leader_election.cc:288] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:17.311905 1381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:17.312543 1381 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [term 1 LEADER]: Becoming Leader. State: Replica: f1eca4053e804dd7a5cea4404352bb56, State: Running, Role: LEADER
I20260620 23:10:17.313076 1381 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1eca4053e804dd7a5cea4404352bb56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } }
I20260620 23:10:17.313761 1371 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:17.315634 1382 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1eca4053e804dd7a5cea4404352bb56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1eca4053e804dd7a5cea4404352bb56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } } }
I20260620 23:10:17.315658 1383 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1eca4053e804dd7a5cea4404352bb56. Latest consensus state: current_term: 1 leader_uuid: "f1eca4053e804dd7a5cea4404352bb56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1eca4053e804dd7a5cea4404352bb56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36419 } } }
I20260620 23:10:17.316075 1383 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:17.316075 1382 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:17.319417 1388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:17.323868 1388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:17.332572 1388 catalog_manager.cc:1347] Generated new cluster ID: ec44082d65e34fa6aa766bc3742777be
I20260620 23:10:17.332701 1388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:17.357069 1388 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:17.358390 1388 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:17.365168 1388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1eca4053e804dd7a5cea4404352bb56: Generated new TSK 0
I20260620 23:10:17.365953 1388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:17.378203 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1304
W20260620 23:10:17.379360 1366 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.196:50185
W20260620 23:10:17.380043 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1304: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:17.380162 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 1304
W20260620 23:10:17.393445 1289 connection.cc:535] client connection to 127.0.0.1:36419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36419): Connection reset by peer (error 104)
2026-06-20T23:10:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1768 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:17 dist-test-slave-7g8g krb5kdc[1402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:17 dist-test-slave-7g8g krb5kdc[1402](info): set up 2 sockets
Jun 20 23:10:17 dist-test-slave-7g8g krb5kdc[1402](info): commencing operation
krb5kdc: starting...
W20260620 23:10:18.399020 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:18 dist-test-slave-7g8g krb5kdc[1402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:10:18.629726 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:18.943102 1415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:18.943379 1415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:18.943506 1415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:18.950618 1415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:18.950727 1415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:18.950806 1415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:18.950889 1415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:18.950966 1415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:18.960824 1415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:18.962075 1415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:18.963369 1415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:18.974478 1420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:18.974504 1421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:18.974439 1423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:18.975174 1415 server_base.cc:1034] running on GCE node
I20260620 23:10:18.975867 1415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:18.977160 1415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:18.978348 1415 hybrid_clock.cc:648] HybridClock initialized: now 1781997018978300 us; error 39 us; skew 500 ppm
I20260620 23:10:18.978667 1415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:18 dist-test-slave-7g8g krb5kdc[1402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:18.981487 1415 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:10:18.982957 1415 webserver.cc:460] Webserver started at http://127.0.0.1:46531/ using document root <none> and password file <none>
I20260620 23:10:18.983604 1415 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:18.983717 1415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:18.983999 1415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:18.986841 1415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "5b7efa586d6e475d930bac8b91f31a4b"
format_stamp: "Formatted at 2026-06-20 23:10:18 on dist-test-slave-7g8g"
server_key: "3c771ace25bad87bc5c4f3e2ad17c908"
server_key_iv: "bb2f698dbdbe6ddc929c750361d54252"
server_key_version: "encryptionkey@0"
I20260620 23:10:18.987579 1415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "5b7efa586d6e475d930bac8b91f31a4b"
format_stamp: "Formatted at 2026-06-20 23:10:18 on dist-test-slave-7g8g"
server_key: "3c771ace25bad87bc5c4f3e2ad17c908"
server_key_iv: "bb2f698dbdbe6ddc929c750361d54252"
server_key_version: "encryptionkey@0"
I20260620 23:10:18.992461 1415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:10:18.995692 1430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:18.996906 1415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:10:18.997047 1415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "5b7efa586d6e475d930bac8b91f31a4b"
format_stamp: "Formatted at 2026-06-20 23:10:18 on dist-test-slave-7g8g"
server_key: "3c771ace25bad87bc5c4f3e2ad17c908"
server_key_iv: "bb2f698dbdbe6ddc929c750361d54252"
server_key_version: "encryptionkey@0"
I20260620 23:10:18.997211 1415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:19.002609 1415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:19.006181 1415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:19.006438 1415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:19.022957 1415 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34473
I20260620 23:10:19.023003 1481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34473 every 8 connection(s)
I20260620 23:10:19.024279 1415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:19.027788 1482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:19.030692 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 1415
I20260620 23:10:19.030848 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:19.031270 12297 external_mini_cluster.cc:1428] Setting key 165d30e40f90f251efeed9c8873de322
I20260620 23:10:19.039657 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:19.033042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:55497 (local address 127.0.0.1:34473)
0620 23:10:19.033748 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:19.033763 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:19.033808 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:10:19.035929 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:19.035957 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:19.036431 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:19.036851 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:19.037779 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:19.037787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:19.037807 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:19.037827 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:19.037894 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:19.037918 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:19.038138 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:19.038281 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 23:10:19.038464 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":272,"threads_started":1}
I20260620 23:10:19.040401 1482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b: Bootstrap starting.
I20260620 23:10:19.043061 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:19.043474 1482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:19.044626 1482 log.cc:824] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:19.047704 1482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b: No bootstrap required, opened a new log
I20260620 23:10:19.049988 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:19.047870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:39017 (local address 127.0.0.1:34473)
0620 23:10:19.048027 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:19.048033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:19.048316 (+ 283us) server_negotiation.cc:407] Connection header received
0620 23:10:19.048440 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:19.048445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:19.048498 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:19.048596 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:19.049400 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:19.049405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:19.049410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:19.049431 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:19.049479 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:19.049502 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:19.049612 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:19.049665 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 23:10:19.049717 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:10:19.055570 1482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7efa586d6e475d930bac8b91f31a4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260620 23:10:19.055903 1482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:19.055974 1482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7efa586d6e475d930bac8b91f31a4b, State: Initialized, Role: FOLLOWER
I20260620 23:10:19.056452 1482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7efa586d6e475d930bac8b91f31a4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260620 23:10:19.056674 1482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:19.056816 1482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:19.056990 1482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:19.059149 1482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7efa586d6e475d930bac8b91f31a4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260620 23:10:19.059605 1482 leader_election.cc:302] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b7efa586d6e475d930bac8b91f31a4b; no voters:
I20260620 23:10:19.060106 1482 leader_election.cc:288] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:19.060225 1492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:19.060807 1492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [term 1 LEADER]: Becoming Leader. State: Replica: 5b7efa586d6e475d930bac8b91f31a4b, State: Running, Role: LEADER
I20260620 23:10:19.061283 1492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7efa586d6e475d930bac8b91f31a4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } }
I20260620 23:10:19.061939 1482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:19.064024 1493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b7efa586d6e475d930bac8b91f31a4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7efa586d6e475d930bac8b91f31a4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } } }
I20260620 23:10:19.064402 1493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:19.065433 1494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b7efa586d6e475d930bac8b91f31a4b. Latest consensus state: current_term: 1 leader_uuid: "5b7efa586d6e475d930bac8b91f31a4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7efa586d6e475d930bac8b91f31a4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34473 } } }
I20260620 23:10:19.065725 1494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:19.066886 1497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:19.071588 1497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:19.080652 1497 catalog_manager.cc:1347] Generated new cluster ID: 2b8b10c8d0db49f7974b0615f099cc83
I20260620 23:10:19.080762 1497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:19.089327 1497 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:19.089708 1497 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:19.096422 1497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b7efa586d6e475d930bac8b91f31a4b: Generated new TSK 0
I20260620 23:10:19.097435 1497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:19.103015 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1415
W20260620 23:10:19.104189 1477 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.196:55497
W20260620 23:10:19.104803 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1415: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:19.104964 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 1415
2026-06-20T23:10:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1724 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:19 dist-test-slave-7g8g krb5kdc[1513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:19 dist-test-slave-7g8g krb5kdc[1513](info): set up 2 sockets
Jun 20 23:10:19 dist-test-slave-7g8g krb5kdc[1513](info): commencing operation
krb5kdc: starting...
W20260620 23:10:20.107117 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:20 dist-test-slave-7g8g krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:10:20.341878 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:35181
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:35181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:20.652736 1526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:20.653012 1526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:20.653100 1526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:20.660230 1526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:20.660347 1526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:20.660398 1526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:20.660456 1526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:20.660498 1526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:20.670089 1526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:35181
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:35181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:20.671314 1526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:20.672555 1526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:20.683400 1532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:20.683452 1531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:20.683477 1534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:20.684412 1526 server_base.cc:1034] running on GCE node
I20260620 23:10:20.685153 1526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:20.686430 1526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:20.687618 1526 hybrid_clock.cc:648] HybridClock initialized: now 1781997020687569 us; error 38 us; skew 500 ppm
I20260620 23:10:20.687947 1526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:20 dist-test-slave-7g8g krb5kdc[1513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997020, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:20.694921 1526 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:10:20.696364 1526 webserver.cc:460] Webserver started at http://127.12.2.126:35915/ using document root <none> and password file <none>
I20260620 23:10:20.696998 1526 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:20.697095 1526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:20.697345 1526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:20.700245 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "a54e00ed67684a4fa6dd7e10ba62dc3f"
format_stamp: "Formatted at 2026-06-20 23:10:20 on dist-test-slave-7g8g"
server_key: "c78f802376c7a9b3eae9a2c8b6985d8c"
server_key_iv: "c4ebffc9e2034255624fd65add820a3c"
server_key_version: "encryptionkey@0"
I20260620 23:10:20.700940 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "a54e00ed67684a4fa6dd7e10ba62dc3f"
format_stamp: "Formatted at 2026-06-20 23:10:20 on dist-test-slave-7g8g"
server_key: "c78f802376c7a9b3eae9a2c8b6985d8c"
server_key_iv: "c4ebffc9e2034255624fd65add820a3c"
server_key_version: "encryptionkey@0"
I20260620 23:10:20.705827 1526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:10:20.709223 1541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:20.710609 1526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:10:20.710776 1526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "a54e00ed67684a4fa6dd7e10ba62dc3f"
format_stamp: "Formatted at 2026-06-20 23:10:20 on dist-test-slave-7g8g"
server_key: "c78f802376c7a9b3eae9a2c8b6985d8c"
server_key_iv: "c4ebffc9e2034255624fd65add820a3c"
server_key_version: "encryptionkey@0"
I20260620 23:10:20.711007 1526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:20.737911 1526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:20.741654 1526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:20.741909 1526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:20.759123 1526 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:35181
I20260620 23:10:20.759187 1592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:35181 every 8 connection(s)
I20260620 23:10:20.760438 1526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:20.763859 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 1526
I20260620 23:10:20.764043 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:20.764211 1593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:20.764591 12297 external_mini_cluster.cc:1428] Setting key eda5aa095ced8399c0c388e29cb277a6
I20260620 23:10:20.778257 1593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f: Bootstrap starting.
Jun 20 23:10:20 dist-test-slave-7g8g krb5kdc[1513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:10:20.781607 1593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:20.782682 1593 log.cc:824] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:20.785426 1593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f: No bootstrap required, opened a new log
I20260620 23:10:20.786863 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:20.766244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46779 (local address 127.12.2.126:35181)
0620 23:10:20.767002 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:20.767016 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:20.767057 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:10:20.768752 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:20.768773 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:20.769149 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:20.769455 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:20.771150 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:20.772392 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:20.774010 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:20.774709 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:20.774843 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:20.779614 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:20.779627 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:20.779642 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:20.779704 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:20.782621 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:20.783478 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:20.783485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:20.783494 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:20.783655 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:20.784106 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:20.784110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:20.784112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:20.784605 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:20.784906 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:20.785377 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 23:10:20.785873 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":308,"threads_started":1}
I20260620 23:10:20.790865 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:20.793752 1593 raft_consensus.cc:348] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54e00ed67684a4fa6dd7e10ba62dc3f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35181 } }
I20260620 23:10:20.794215 1593 raft_consensus.cc:374] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:20.794303 1593 raft_consensus.cc:729] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a54e00ed67684a4fa6dd7e10ba62dc3f, State: Initialized, Role: FOLLOWER
I20260620 23:10:20.794955 1593 consensus_queue.cc:260] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54e00ed67684a4fa6dd7e10ba62dc3f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35181 } }
I20260620 23:10:20.795311 1593 raft_consensus.cc:388] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:20.795440 1593 raft_consensus.cc:482] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:20.795605 1593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:20.797780 1593 raft_consensus.cc:504] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54e00ed67684a4fa6dd7e10ba62dc3f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35181 } }
I20260620 23:10:20.798327 1593 leader_election.cc:302] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a54e00ed67684a4fa6dd7e10ba62dc3f; no voters:
I20260620 23:10:20.798985 1593 leader_election.cc:288] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:20.799093 1603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:20.800287 1603 raft_consensus.cc:686] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [term 1 LEADER]: Becoming Leader. State: Replica: a54e00ed67684a4fa6dd7e10ba62dc3f, State: Running, Role: LEADER
I20260620 23:10:20.800806 1603 consensus_queue.cc:237] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54e00ed67684a4fa6dd7e10ba62dc3f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35181 } }
I20260620 23:10:20.801294 1593 sys_catalog.cc:564] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:20.803401 1604 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a54e00ed67684a4fa6dd7e10ba62dc3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54e00ed67684a4fa6dd7e10ba62dc3f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35181 } } }
I20260620 23:10:20.803785 1604 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:20.805574 1605 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a54e00ed67684a4fa6dd7e10ba62dc3f. Latest consensus state: current_term: 1 leader_uuid: "a54e00ed67684a4fa6dd7e10ba62dc3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54e00ed67684a4fa6dd7e10ba62dc3f" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35181 } } }
I20260620 23:10:20.805893 1605 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:20.807278 1608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:20.811528 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:20.795020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46337 (local address 127.12.2.126:35181)
0620 23:10:20.795202 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:20.795208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:20.795544 (+ 336us) server_negotiation.cc:407] Connection header received
0620 23:10:20.795718 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:20.795722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:20.795780 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:20.795865 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:20.797184 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:20.797914 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:20.799671 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:20.800315 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:20.800423 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:20.803765 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:20.803776 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:20.803782 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:20.803825 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:20.807760 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:20.808725 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:20.808731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:20.808735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:20.808887 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:20.809498 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:20.809502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:20.809506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:20.809944 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:20.810255 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:20.810965 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 23:10:20.811163 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:10:20.813608 1608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:20.821940 1608 catalog_manager.cc:1347] Generated new cluster ID: dd5b7da24f344e66a10d34c57aea54cf
I20260620 23:10:20.822050 1608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:20.838416 1608 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:20.839679 1608 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:20.851186 1608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a54e00ed67684a4fa6dd7e10ba62dc3f: Generated new TSK 0
I20260620 23:10:20.852325 1608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:20.868933 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1526
I20260620 23:10:20.870828 1588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:20.961858 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 1526
W20260620 23:10:20.975414 1511 connection.cc:535] client connection to 127.12.2.126:35181 recv error: Network error: failed to read from TLS socket (remote: 127.12.2.126:35181): Connection reset by peer (error 104)
2026-06-20T23:10:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1856 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:21 dist-test-slave-7g8g krb5kdc[1625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:21 dist-test-slave-7g8g krb5kdc[1625](info): set up 2 sockets
Jun 20 23:10:21 dist-test-slave-7g8g krb5kdc[1625](info): commencing operation
krb5kdc: starting...
W20260620 23:10:21.967063 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:22 dist-test-slave-7g8g krb5kdc[1625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:10:22.198799 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:45305
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:39175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:45305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:22.511888 1638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:22.512161 1638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:22.512290 1638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:22.519364 1638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:22.519474 1638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:22.519549 1638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:22.519640 1638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:22.519701 1638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:22.529708 1638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:39175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:45305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:45305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:22.531008 1638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:22.532298 1638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:22.543555 1646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:22.543704 1644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:22.543949 1643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:22.544878 1638 server_base.cc:1034] running on GCE node
I20260620 23:10:22.545641 1638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:22.547011 1638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:22.548205 1638 hybrid_clock.cc:648] HybridClock initialized: now 1781997022548158 us; error 44 us; skew 500 ppm
I20260620 23:10:22.548524 1638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:22 dist-test-slave-7g8g krb5kdc[1625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997022, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:22.551383 1638 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:10:22.552806 1638 webserver.cc:460] Webserver started at http://127.12.2.126:42387/ using document root <none> and password file <none>
I20260620 23:10:22.553488 1638 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:22.553613 1638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:22.553889 1638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:22.556823 1638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "f0319047848c4e628a8405ad0cb3d00d"
format_stamp: "Formatted at 2026-06-20 23:10:22 on dist-test-slave-7g8g"
server_key: "e2bb8bce5443bb79d71723d4d1b793c8"
server_key_iv: "db551ce409b12aea27991ff4aa4369ed"
server_key_version: "encryptionkey@0"
I20260620 23:10:22.557543 1638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "f0319047848c4e628a8405ad0cb3d00d"
format_stamp: "Formatted at 2026-06-20 23:10:22 on dist-test-slave-7g8g"
server_key: "e2bb8bce5443bb79d71723d4d1b793c8"
server_key_iv: "db551ce409b12aea27991ff4aa4369ed"
server_key_version: "encryptionkey@0"
I20260620 23:10:22.562455 1638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:10:22.565793 1653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:22.567066 1638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:22.567215 1638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "f0319047848c4e628a8405ad0cb3d00d"
format_stamp: "Formatted at 2026-06-20 23:10:22 on dist-test-slave-7g8g"
server_key: "e2bb8bce5443bb79d71723d4d1b793c8"
server_key_iv: "db551ce409b12aea27991ff4aa4369ed"
server_key_version: "encryptionkey@0"
I20260620 23:10:22.567382 1638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:22.590114 1638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:22.594039 1638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:22.594297 1638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:22.611179 1638 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:45305
I20260620 23:10:22.611202 1704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:45305 every 8 connection(s)
I20260620 23:10:22.612488 1638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:22.616266 1705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:22.619789 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 1638
I20260620 23:10:22.619971 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:22.620505 12297 external_mini_cluster.cc:1428] Setting key c891a1e47e699153fd3d09fefb9db9e2
I20260620 23:10:22.628326 1705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d: Bootstrap starting.
I20260620 23:10:22.632011 1705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:22.633261 1705 log.cc:824] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:22.636296 1708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:22.622438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46113 (local address 127.12.2.126:45305)
0620 23:10:22.623337 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:22.623352 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:22.623737 (+ 385us) server_negotiation.cc:407] Connection header received
0620 23:10:22.626471 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:22.626499 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:22.627053 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:22.627478 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:22.627989 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:22.629964 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:22.631373 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:22.632407 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:22.632609 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:22.633680 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:22.633690 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:22.633712 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:22.634348 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:22.634431 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:22.634462 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:22.634614 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:22.634858 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 23:10:22.635040 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":417,"threads_started":1}
I20260620 23:10:22.636462 1705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d: No bootstrap required, opened a new log
I20260620 23:10:22.640295 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:22.646111 1705 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0319047848c4e628a8405ad0cb3d00d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45305 } }
I20260620 23:10:22.646576 1705 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:22.646672 1705 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0319047848c4e628a8405ad0cb3d00d, State: Initialized, Role: FOLLOWER
I20260620 23:10:22.647271 1705 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0319047848c4e628a8405ad0cb3d00d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45305 } }
I20260620 23:10:22.647493 1705 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:22.647624 1705 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:22.647774 1705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:22.649936 1705 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0319047848c4e628a8405ad0cb3d00d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45305 } }
I20260620 23:10:22.650520 1705 leader_election.cc:302] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0319047848c4e628a8405ad0cb3d00d; no voters:
I20260620 23:10:22.651240 1705 leader_election.cc:288] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:22.651340 1715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:22.651654 1708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:22.645344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51823 (local address 127.12.2.126:45305)
0620 23:10:22.645514 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:22.645520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:22.645917 (+ 397us) server_negotiation.cc:407] Connection header received
0620 23:10:22.646067 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:22.646071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:22.646155 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:22.646262 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:22.646732 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:22.647629 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:22.648932 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:22.649385 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:22.649496 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:22.650635 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:22.650639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:22.650643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:22.650660 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:22.650720 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:22.650739 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:22.650836 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:22.651094 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 23:10:22.651157 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:10:22.653431 1705 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:22.657778 1715 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [term 1 LEADER]: Becoming Leader. State: Replica: f0319047848c4e628a8405ad0cb3d00d, State: Running, Role: LEADER
I20260620 23:10:22.658313 1715 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0319047848c4e628a8405ad0cb3d00d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45305 } }
I20260620 23:10:22.661024 1716 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0319047848c4e628a8405ad0cb3d00d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0319047848c4e628a8405ad0cb3d00d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45305 } } }
I20260620 23:10:22.661175 1717 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0319047848c4e628a8405ad0cb3d00d. Latest consensus state: current_term: 1 leader_uuid: "f0319047848c4e628a8405ad0cb3d00d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0319047848c4e628a8405ad0cb3d00d" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45305 } } }
I20260620 23:10:22.661422 1716 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:22.661449 1717 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d [sys.catalog]: This master's current role is: LEADER
W20260620 23:10:22.665740 1729 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:10:22.665853 1729 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:10:22.665977 1730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:22.668131 1730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:22.676963 1730 catalog_manager.cc:1347] Generated new cluster ID: a91886b3206f40edb0b4a67f69b692d8
I20260620 23:10:22.677083 1730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:22.690717 1730 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:22.692494 1730 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:22.715461 1730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0319047848c4e628a8405ad0cb3d00d: Generated new TSK 0
I20260620 23:10:22.716557 1730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:22.726202 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1638
W20260620 23:10:22.727632 1700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46113
W20260620 23:10:22.728389 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1638: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:22.728564 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 1638
W20260620 23:10:22.741958 1623 connection.cc:535] client connection to 127.12.2.126:45305 recv error: Network error: failed to read from TLS socket (remote: 127.12.2.126:45305): Connection reset by peer (error 104)
2026-06-20T23:10:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1766 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:22 dist-test-slave-7g8g krb5kdc[1736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:22 dist-test-slave-7g8g krb5kdc[1736](info): set up 2 sockets
Jun 20 23:10:22 dist-test-slave-7g8g krb5kdc[1736](info): commencing operation
krb5kdc: starting...
W20260620 23:10:23.738984 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:23 dist-test-slave-7g8g krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:10:23.975729 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43037
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:43037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:24.293816 1749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:24.294098 1749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:24.294188 1749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:24.301350 1749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:24.301463 1749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:24.301517 1749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:24.301586 1749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:24.301628 1749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:24.311352 1749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:43037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:43037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:24.313161 1749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:24.314797 1749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:24.327672 1754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:24.327744 1755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:24.327709 1757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:24.328647 1749 server_base.cc:1034] running on GCE node
I20260620 23:10:24.329382 1749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:24.330761 1749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:24.331960 1749 hybrid_clock.cc:648] HybridClock initialized: now 1781997024331911 us; error 40 us; skew 500 ppm
I20260620 23:10:24.332288 1749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:24 dist-test-slave-7g8g krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997024, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:24.335287 1749 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:10:24.336795 1749 webserver.cc:460] Webserver started at http://127.12.2.126:36823/ using document root <none> and password file <none>
I20260620 23:10:24.337436 1749 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:24.337536 1749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:24.337832 1749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:24.340768 1749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "26afc69aed03436b83d0938ae430c284"
format_stamp: "Formatted at 2026-06-20 23:10:24 on dist-test-slave-7g8g"
server_key: "1148b90e7308b57f5f7dfae0913ad4f7"
server_key_iv: "a34b49949723cbc7c6e969b6aec78fea"
server_key_version: "encryptionkey@0"
I20260620 23:10:24.341472 1749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "26afc69aed03436b83d0938ae430c284"
format_stamp: "Formatted at 2026-06-20 23:10:24 on dist-test-slave-7g8g"
server_key: "1148b90e7308b57f5f7dfae0913ad4f7"
server_key_iv: "a34b49949723cbc7c6e969b6aec78fea"
server_key_version: "encryptionkey@0"
I20260620 23:10:24.346365 1749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:10:24.349810 1764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:24.351238 1749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:24.351395 1749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "26afc69aed03436b83d0938ae430c284"
format_stamp: "Formatted at 2026-06-20 23:10:24 on dist-test-slave-7g8g"
server_key: "1148b90e7308b57f5f7dfae0913ad4f7"
server_key_iv: "a34b49949723cbc7c6e969b6aec78fea"
server_key_version: "encryptionkey@0"
I20260620 23:10:24.351578 1749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:24.357214 1749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:24.361037 1749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:24.361296 1749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:24.378407 1749 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:43037
I20260620 23:10:24.378450 1815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:43037 every 8 connection(s)
I20260620 23:10:24.379772 1749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:24.383381 1816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:24.388706 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 1749
I20260620 23:10:24.388857 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:24.389259 12297 external_mini_cluster.cc:1428] Setting key 3b62932459229f557557d0cabb10fedd
I20260620 23:10:24.394409 1816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284: Bootstrap starting.
I20260620 23:10:24.397683 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:24.391013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54555 (local address 127.12.2.126:43037)
0620 23:10:24.391836 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:24.391851 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:24.391907 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:10:24.394080 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:24.394117 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:24.394529 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:24.394963 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:24.395860 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:24.395868 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:24.395883 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:24.395904 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:24.395974 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:24.396001 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:24.396251 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:24.396371 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 23:10:24.396554 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":336,"threads_started":1}
I20260620 23:10:24.398216 1816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:24.399252 1816 log.cc:824] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:24.401171 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:24.402011 1816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284: No bootstrap required, opened a new log
I20260620 23:10:24.407523 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:24.405176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50711 (local address 127.12.2.126:43037)
0620 23:10:24.405345 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:24.405351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:24.405732 (+ 381us) server_negotiation.cc:407] Connection header received
0620 23:10:24.405867 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:24.405873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:24.405938 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:24.406037 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:24.406878 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:24.406883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:24.406888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:24.406906 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:24.406976 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:24.406997 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:24.407122 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:24.407188 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 23:10:24.407246 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:10:24.410034 1816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc69aed03436b83d0938ae430c284" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43037 } }
I20260620 23:10:24.410389 1816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:24.410490 1816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26afc69aed03436b83d0938ae430c284, State: Initialized, Role: FOLLOWER
I20260620 23:10:24.411109 1816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc69aed03436b83d0938ae430c284" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43037 } }
I20260620 23:10:24.411314 1816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:24.411409 1816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:24.411528 1816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:24.413542 1816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc69aed03436b83d0938ae430c284" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43037 } }
I20260620 23:10:24.413964 1816 leader_election.cc:302] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26afc69aed03436b83d0938ae430c284; no voters:
I20260620 23:10:24.414462 1816 leader_election.cc:288] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:24.414568 1826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:24.415496 1826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [term 1 LEADER]: Becoming Leader. State: Replica: 26afc69aed03436b83d0938ae430c284, State: Running, Role: LEADER
I20260620 23:10:24.416069 1826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc69aed03436b83d0938ae430c284" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43037 } }
I20260620 23:10:24.416497 1816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:24.418695 1828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26afc69aed03436b83d0938ae430c284. Latest consensus state: current_term: 1 leader_uuid: "26afc69aed03436b83d0938ae430c284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc69aed03436b83d0938ae430c284" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43037 } } }
I20260620 23:10:24.419394 1828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:24.418722 1827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26afc69aed03436b83d0938ae430c284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26afc69aed03436b83d0938ae430c284" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 43037 } } }
I20260620 23:10:24.419966 1827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:24.421620 1833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:24.426007 1833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:24.434087 1833 catalog_manager.cc:1347] Generated new cluster ID: fdefd5eb496d4300933ef0697ad62056
I20260620 23:10:24.434194 1833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:24.443141 1833 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:24.443509 1833 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:24.452471 1833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26afc69aed03436b83d0938ae430c284: Generated new TSK 0
I20260620 23:10:24.453254 1833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:24.459218 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1749
W20260620 23:10:24.460327 1811 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54555
W20260620 23:10:24.461069 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1749: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:24.461229 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 1749
2026-06-20T23:10:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1732 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:24 dist-test-slave-7g8g krb5kdc[1847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:24 dist-test-slave-7g8g krb5kdc[1847](info): set up 2 sockets
Jun 20 23:10:24 dist-test-slave-7g8g krb5kdc[1847](info): commencing operation
krb5kdc: starting...
W20260620 23:10:25.471092 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:25 dist-test-slave-7g8g krb5kdc[1847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:10:25.708031 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:41083
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:41083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:26.018209 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:26.018520 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:26.018653 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:26.026067 1860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:26.026185 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:26.026238 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:26.026319 1860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:26.026381 1860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:26.036370 1860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:41083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:41083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:26.037647 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:26.038920 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:26.050041 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:26.049970 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:26.050015 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:26.050858 1860 server_base.cc:1034] running on GCE node
I20260620 23:10:26.051648 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:26.053107 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:26.054306 1860 hybrid_clock.cc:648] HybridClock initialized: now 1781997026054259 us; error 45 us; skew 500 ppm
I20260620 23:10:26.054641 1860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:26 dist-test-slave-7g8g krb5kdc[1847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997026, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:26.062152 1860 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:10:26.063704 1860 webserver.cc:460] Webserver started at http://127.12.2.126:37387/ using document root <none> and password file <none>
I20260620 23:10:26.064396 1860 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:26.064514 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:26.064795 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:26.067821 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "eb6e78a8f9e6413782d3e8ddd9b576c0"
format_stamp: "Formatted at 2026-06-20 23:10:26 on dist-test-slave-7g8g"
server_key: "7daef554a968811a95d845293fe010f6"
server_key_iv: "a22d8146bfa440de822a7a94e72c5fd1"
server_key_version: "encryptionkey@0"
I20260620 23:10:26.068557 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "eb6e78a8f9e6413782d3e8ddd9b576c0"
format_stamp: "Formatted at 2026-06-20 23:10:26 on dist-test-slave-7g8g"
server_key: "7daef554a968811a95d845293fe010f6"
server_key_iv: "a22d8146bfa440de822a7a94e72c5fd1"
server_key_version: "encryptionkey@0"
I20260620 23:10:26.073624 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:10:26.077093 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:26.078517 1860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:10:26.078703 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "eb6e78a8f9e6413782d3e8ddd9b576c0"
format_stamp: "Formatted at 2026-06-20 23:10:26 on dist-test-slave-7g8g"
server_key: "7daef554a968811a95d845293fe010f6"
server_key_iv: "a22d8146bfa440de822a7a94e72c5fd1"
server_key_version: "encryptionkey@0"
I20260620 23:10:26.078895 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:26.100382 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:26.104228 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:26.104485 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:26.121311 1860 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:41083
I20260620 23:10:26.121333 1926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:41083 every 8 connection(s)
I20260620 23:10:26.122632 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:26.126266 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:26.129904 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 1860
I20260620 23:10:26.130101 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:26.130601 12297 external_mini_cluster.cc:1428] Setting key 5784df7e8342ab30bff26f0315ca3adc
I20260620 23:10:26.138597 1927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0: Bootstrap starting.
I20260620 23:10:26.142580 1927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:26.143615 1927 log.cc:824] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:10:26 dist-test-slave-7g8g krb5kdc[1847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:10:26.146763 1927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0: No bootstrap required, opened a new log
I20260620 23:10:26.153848 1930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:26.132351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36163 (local address 127.12.2.126:41083)
0620 23:10:26.133036 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:26.133048 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:26.133088 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:10:26.134686 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:26.134709 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:26.135176 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:26.135516 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:26.137217 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:26.138495 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:26.140287 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:26.141323 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:26.141490 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:26.146502 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:26.146510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:26.146524 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:26.146565 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:26.149545 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:26.150348 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:26.150354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:26.150364 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:26.150528 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:26.150915 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:26.150918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:26.150920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:26.151473 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:26.151691 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:26.152150 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 23:10:26.152492 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":302,"threads_started":1}
I20260620 23:10:26.155843 1927 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6e78a8f9e6413782d3e8ddd9b576c0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41083 } }
I20260620 23:10:26.156278 1927 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:26.156383 1927 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6e78a8f9e6413782d3e8ddd9b576c0, State: Initialized, Role: FOLLOWER
I20260620 23:10:26.156983 1927 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6e78a8f9e6413782d3e8ddd9b576c0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41083 } }
I20260620 23:10:26.157274 1927 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:26.157409 1927 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:26.157502 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:26.157567 1927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:26.160018 1927 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6e78a8f9e6413782d3e8ddd9b576c0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41083 } }
I20260620 23:10:26.160671 1927 leader_election.cc:302] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb6e78a8f9e6413782d3e8ddd9b576c0; no voters:
I20260620 23:10:26.161283 1927 leader_election.cc:288] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:26.161365 1936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:26.163817 1927 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:26.168241 1936 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [term 1 LEADER]: Becoming Leader. State: Replica: eb6e78a8f9e6413782d3e8ddd9b576c0, State: Running, Role: LEADER
I20260620 23:10:26.168869 1936 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6e78a8f9e6413782d3e8ddd9b576c0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41083 } }
I20260620 23:10:26.172029 1937 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6e78a8f9e6413782d3e8ddd9b576c0. Latest consensus state: current_term: 1 leader_uuid: "eb6e78a8f9e6413782d3e8ddd9b576c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6e78a8f9e6413782d3e8ddd9b576c0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41083 } } }
I20260620 23:10:26.172443 1937 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:26.178514 1939 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb6e78a8f9e6413782d3e8ddd9b576c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6e78a8f9e6413782d3e8ddd9b576c0" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 41083 } } }
I20260620 23:10:26.179286 1939 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:26.181375 1930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:26.161642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45991 (local address 127.12.2.126:41083)
0620 23:10:26.161822 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:26.161828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:26.167642 (+ 5814us) server_negotiation.cc:407] Connection header received
0620 23:10:26.167730 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:26.167735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:26.167804 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:26.167891 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:26.169353 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:26.170071 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:26.172373 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:26.172996 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:26.173083 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:26.176025 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:26.176029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:26.176033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:26.176065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:26.178834 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:26.179504 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:26.179507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:26.179510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:26.179623 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:26.179980 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:26.179986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:26.179989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:26.180284 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:26.180491 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:26.180832 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 23:10:26.181040 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:10:26.183343 1944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:26.189136 1944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:26.197999 1944 catalog_manager.cc:1347] Generated new cluster ID: 0e9f5b6845ec4d1cb17f788f10ff77fc
I20260620 23:10:26.198124 1944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:26.222488 1944 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:26.224498 1944 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:26.235702 1944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb6e78a8f9e6413782d3e8ddd9b576c0: Generated new TSK 0
I20260620 23:10:26.236851 1944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:26.247659 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1860
I20260620 23:10:26.248754 1922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:26.338830 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 1860
2026-06-20T23:10:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:26 dist-test-slave-7g8g krb5kdc[1959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:26 dist-test-slave-7g8g krb5kdc[1959](info): set up 2 sockets
Jun 20 23:10:26 dist-test-slave-7g8g krb5kdc[1959](info): commencing operation
krb5kdc: starting...
W20260620 23:10:27.359089 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:27 dist-test-slave-7g8g krb5kdc[1959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:10:27.606132 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:32769
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:32769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:27.918533 1972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:27.918799 1972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:27.918898 1972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:27.925911 1972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:27.926010 1972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:27.926062 1972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:27.926123 1972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:27.926174 1972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:27.935810 1972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:32769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:32769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:27.937070 1972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:27.938298 1972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:27.949127 1978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:27.949229 1977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:27.949164 1980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:27.950079 1972 server_base.cc:1034] running on GCE node
I20260620 23:10:27.950801 1972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:27.952145 1972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:27.953334 1972 hybrid_clock.cc:648] HybridClock initialized: now 1781997027953279 us; error 44 us; skew 500 ppm
I20260620 23:10:27.953641 1972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:27 dist-test-slave-7g8g krb5kdc[1959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997027, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:27.956552 1972 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:10:27.958127 1972 webserver.cc:460] Webserver started at http://127.12.2.126:42431/ using document root <none> and password file <none>
I20260620 23:10:27.958732 1972 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:27.958827 1972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:27.959121 1972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:27.961922 1972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "91f31abe3dcf40628cd165e72642dee2"
format_stamp: "Formatted at 2026-06-20 23:10:27 on dist-test-slave-7g8g"
server_key: "b43e90218b44551717a5fd597f29b00a"
server_key_iv: "f053a9417837a510e8008604f6e3443e"
server_key_version: "encryptionkey@0"
I20260620 23:10:27.962601 1972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "91f31abe3dcf40628cd165e72642dee2"
format_stamp: "Formatted at 2026-06-20 23:10:27 on dist-test-slave-7g8g"
server_key: "b43e90218b44551717a5fd597f29b00a"
server_key_iv: "f053a9417837a510e8008604f6e3443e"
server_key_version: "encryptionkey@0"
I20260620 23:10:27.967494 1972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 23:10:27.970685 1987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:27.971966 1972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:27.972113 1972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "91f31abe3dcf40628cd165e72642dee2"
format_stamp: "Formatted at 2026-06-20 23:10:27 on dist-test-slave-7g8g"
server_key: "b43e90218b44551717a5fd597f29b00a"
server_key_iv: "f053a9417837a510e8008604f6e3443e"
server_key_version: "encryptionkey@0"
I20260620 23:10:27.972275 1972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:27.987412 1972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:27.991032 1972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:27.991267 1972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:28.007875 1972 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:32769
I20260620 23:10:28.007895 2038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:32769 every 8 connection(s)
I20260620 23:10:28.009178 1972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:28.012658 2039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:28.017607 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 1972
I20260620 23:10:28.017750 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:28.018141 12297 external_mini_cluster.cc:1428] Setting key 9e14ba0ba16e7f3d3d8fd77355039a20
I20260620 23:10:28.023005 2039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2: Bootstrap starting.
I20260620 23:10:28.026422 2039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:28.027503 2039 log.cc:824] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:28.030714 2039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2: No bootstrap required, opened a new log
I20260620 23:10:28.030905 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:28.019792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46101 (local address 127.12.2.126:32769)
0620 23:10:28.020578 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:28.020592 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:28.020633 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:10:28.022276 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:28.022299 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:28.022690 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:28.023038 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:28.023595 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:28.024864 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:28.026619 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:28.027307 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:28.027433 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:28.028660 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:28.028672 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:28.028691 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:28.029312 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:28.029391 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:28.029414 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:28.029497 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:28.029709 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 23:10:28.029940 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":346,"threads_started":1}
I20260620 23:10:28.034355 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:28.038359 2039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f31abe3dcf40628cd165e72642dee2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32769 } }
I20260620 23:10:28.038810 2039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:28.038911 2039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f31abe3dcf40628cd165e72642dee2, State: Initialized, Role: FOLLOWER
I20260620 23:10:28.039569 2039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f31abe3dcf40628cd165e72642dee2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32769 } }
I20260620 23:10:28.039866 2039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:28.040011 2039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:28.040201 2039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:28.043115 2039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f31abe3dcf40628cd165e72642dee2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32769 } }
I20260620 23:10:28.043736 2039 leader_election.cc:302] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91f31abe3dcf40628cd165e72642dee2; no voters:
I20260620 23:10:28.044358 2039 leader_election.cc:288] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:28.044466 2049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:28.045164 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:28.038436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51123 (local address 127.12.2.126:32769)
0620 23:10:28.038612 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:28.038618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:28.039142 (+ 524us) server_negotiation.cc:407] Connection header received
0620 23:10:28.039344 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:28.039349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:28.039410 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:28.039484 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:28.040029 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:28.040819 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:28.042616 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:28.043099 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:28.043193 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:28.044260 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:28.044269 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:28.044274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:28.044298 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:28.044367 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:28.044388 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:28.044480 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:28.044717 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:10:28.044811 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:10:28.046033 2049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [term 1 LEADER]: Becoming Leader. State: Replica: 91f31abe3dcf40628cd165e72642dee2, State: Running, Role: LEADER
I20260620 23:10:28.046597 2039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:28.046696 2049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f31abe3dcf40628cd165e72642dee2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32769 } }
I20260620 23:10:28.049361 2050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91f31abe3dcf40628cd165e72642dee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f31abe3dcf40628cd165e72642dee2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32769 } } }
I20260620 23:10:28.049734 2050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:28.051115 2051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f31abe3dcf40628cd165e72642dee2. Latest consensus state: current_term: 1 leader_uuid: "91f31abe3dcf40628cd165e72642dee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f31abe3dcf40628cd165e72642dee2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 32769 } } }
I20260620 23:10:28.051352 2051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:28.051928 2056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:28.056221 2056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:28.064993 2056 catalog_manager.cc:1347] Generated new cluster ID: 86cc65c797914c6ba796419e3821c842
I20260620 23:10:28.065110 2056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:28.091552 2056 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:28.093226 2056 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:28.106031 2056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91f31abe3dcf40628cd165e72642dee2: Generated new TSK 0
I20260620 23:10:28.106835 2056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:28.121121 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1972
W20260620 23:10:28.122243 2034 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46101
W20260620 23:10:28.122852 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 1972: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:28.123011 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 1972
2026-06-20T23:10:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1785 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:28 dist-test-slave-7g8g krb5kdc[2070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:28 dist-test-slave-7g8g krb5kdc[2070](info): set up 2 sockets
Jun 20 23:10:28 dist-test-slave-7g8g krb5kdc[2070](info): commencing operation
krb5kdc: starting...
W20260620 23:10:29.130862 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:29 dist-test-slave-7g8g krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:10:29.366676 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:35331
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:35331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:29.677908 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:29.678215 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:29.678351 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:29.685489 2083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:29.685627 2083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:29.685707 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:29.685787 2083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:29.685844 2083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:29.695657 2083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:35331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:35331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:29.696964 2083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:29.698298 2083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:29.709427 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:29.709462 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:29.709491 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:29.710595 2083 server_base.cc:1034] running on GCE node
I20260620 23:10:29.711370 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:29.712715 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:29.713915 2083 hybrid_clock.cc:648] HybridClock initialized: now 1781997029713867 us; error 43 us; skew 500 ppm
I20260620 23:10:29.714232 2083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:29 dist-test-slave-7g8g krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997029, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:29.717175 2083 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:10:29.718547 2083 webserver.cc:460] Webserver started at http://127.12.2.126:42361/ using document root <none> and password file <none>
I20260620 23:10:29.719259 2083 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:29.719373 2083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:29.719686 2083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:29.722541 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "47ef74ebb2ab492b9e931cc7ef7552c2"
format_stamp: "Formatted at 2026-06-20 23:10:29 on dist-test-slave-7g8g"
server_key: "a3b927e6d4b74c07153f6f2e69026b53"
server_key_iv: "8967649c71cbb5ed0af5e607c412402f"
server_key_version: "encryptionkey@0"
I20260620 23:10:29.723312 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "47ef74ebb2ab492b9e931cc7ef7552c2"
format_stamp: "Formatted at 2026-06-20 23:10:29 on dist-test-slave-7g8g"
server_key: "a3b927e6d4b74c07153f6f2e69026b53"
server_key_iv: "8967649c71cbb5ed0af5e607c412402f"
server_key_version: "encryptionkey@0"
I20260620 23:10:29.728287 2083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 23:10:29.731572 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:29.732852 2083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:10:29.733011 2083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "47ef74ebb2ab492b9e931cc7ef7552c2"
format_stamp: "Formatted at 2026-06-20 23:10:29 on dist-test-slave-7g8g"
server_key: "a3b927e6d4b74c07153f6f2e69026b53"
server_key_iv: "8967649c71cbb5ed0af5e607c412402f"
server_key_version: "encryptionkey@0"
I20260620 23:10:29.733186 2083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:29.738523 2083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:29.742491 2083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:29.742753 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:29.758761 2083 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:35331
I20260620 23:10:29.758780 2149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:35331 every 8 connection(s)
I20260620 23:10:29.760133 2083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:29.763759 2150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:29.768954 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 2083
I20260620 23:10:29.769109 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:29.769503 12297 external_mini_cluster.cc:1428] Setting key 89930dccfe9d662d3f15450443284179
I20260620 23:10:29.775058 2150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2: Bootstrap starting.
I20260620 23:10:29.778175 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:29.771304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56863 (local address 127.12.2.126:35331)
0620 23:10:29.772145 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:29.772159 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:29.772212 (+ 53us) server_negotiation.cc:407] Connection header received
0620 23:10:29.774394 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:29.774423 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:29.774845 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:29.775299 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:29.776265 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:29.776274 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:29.776290 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:29.776308 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:29.776378 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:29.776406 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:29.776612 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:29.776763 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 23:10:29.776971 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":327,"threads_started":1}
I20260620 23:10:29.778874 2150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:29.780090 2150 log.cc:824] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:29.781849 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:29.782791 2150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2: No bootstrap required, opened a new log
I20260620 23:10:29.788138 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:29.785897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49289 (local address 127.12.2.126:35331)
0620 23:10:29.786062 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:29.786068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:29.786382 (+ 314us) server_negotiation.cc:407] Connection header received
0620 23:10:29.786508 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:29.786514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:29.786574 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:29.786680 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:29.787517 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:29.787524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:29.787529 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:10:29.787547 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:29.787596 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:10:29.787617 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:29.787727 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:29.787789 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 23:10:29.787845 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:10:29.790809 2150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef74ebb2ab492b9e931cc7ef7552c2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35331 } }
I20260620 23:10:29.791185 2150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:29.791260 2150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ef74ebb2ab492b9e931cc7ef7552c2, State: Initialized, Role: FOLLOWER
I20260620 23:10:29.791786 2150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef74ebb2ab492b9e931cc7ef7552c2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35331 } }
I20260620 23:10:29.791992 2150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:29.792085 2150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:29.792203 2150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:29.794507 2150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef74ebb2ab492b9e931cc7ef7552c2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35331 } }
I20260620 23:10:29.794981 2150 leader_election.cc:302] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47ef74ebb2ab492b9e931cc7ef7552c2; no voters:
I20260620 23:10:29.795539 2150 leader_election.cc:288] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:29.795693 2160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:29.796360 2160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [term 1 LEADER]: Becoming Leader. State: Replica: 47ef74ebb2ab492b9e931cc7ef7552c2, State: Running, Role: LEADER
I20260620 23:10:29.796876 2160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef74ebb2ab492b9e931cc7ef7552c2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35331 } }
I20260620 23:10:29.797478 2150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:29.799599 2162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47ef74ebb2ab492b9e931cc7ef7552c2. Latest consensus state: current_term: 1 leader_uuid: "47ef74ebb2ab492b9e931cc7ef7552c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef74ebb2ab492b9e931cc7ef7552c2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35331 } } }
I20260620 23:10:29.800028 2162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:29.800974 2161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47ef74ebb2ab492b9e931cc7ef7552c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ef74ebb2ab492b9e931cc7ef7552c2" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 35331 } } }
I20260620 23:10:29.801365 2161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:29.803731 2167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:29.809042 2167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:29.818421 2167 catalog_manager.cc:1347] Generated new cluster ID: f1926bd1c5cc432aa3ae040fb9a9c359
I20260620 23:10:29.818527 2167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:29.837241 2167 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:29.837757 2167 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:29.845953 2167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47ef74ebb2ab492b9e931cc7ef7552c2: Generated new TSK 0
I20260620 23:10:29.846908 2167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:29.853437 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 2083
W20260620 23:10:29.854473 2145 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56863
W20260620 23:10:29.855118 12297 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 2083: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:29.855269 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 2083
2026-06-20T23:10:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1730 ms)
[----------] 18 tests from AuthTokenIssuingTest (32178 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:29 dist-test-slave-7g8g krb5kdc[2181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:29 dist-test-slave-7g8g krb5kdc[2181](info): set up 2 sockets
Jun 20 23:10:29 dist-test-slave-7g8g krb5kdc[2181](info): commencing operation
krb5kdc: starting...
W20260620 23:10:30.867287 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:31 dist-test-slave-7g8g krb5kdc[2181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:10:31.114274 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:45695
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:45695,127.12.2.125:42801,127.12.2.124:43979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:31.431852 2194 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:10:31.432160 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:31.432257 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:31.432368 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:31.439432 2194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:31.439527 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:31.439613 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:31.439674 2194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:31.439754 2194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:31.449327 2194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:45695,127.12.2.125:42801,127.12.2.124:43979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:45695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:31.450552 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:31.451833 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:31.462677 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:31.462715 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:31.462692 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:31.463351 2194 server_base.cc:1034] running on GCE node
I20260620 23:10:31.464061 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:31.465353 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:31.466593 2194 hybrid_clock.cc:648] HybridClock initialized: now 1781997031466529 us; error 46 us; skew 500 ppm
I20260620 23:10:31.466917 2194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:31 dist-test-slave-7g8g krb5kdc[2181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:31.474185 2194 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:10:31.475529 2194 webserver.cc:460] Webserver started at http://127.12.2.126:44743/ using document root <none> and password file <none>
I20260620 23:10:31.476224 2194 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:31.476320 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:31.476613 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:31.479476 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "86898c47bcba4648945fa45651864fe6"
format_stamp: "Formatted at 2026-06-20 23:10:31 on dist-test-slave-7g8g"
server_key: "79b79b1c6da06c22c6a7aaa549ab0a64"
server_key_iv: "d58d4dff10e05cd5d3acf2f60ed1f622"
server_key_version: "encryptionkey@0"
I20260620 23:10:31.480209 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "86898c47bcba4648945fa45651864fe6"
format_stamp: "Formatted at 2026-06-20 23:10:31 on dist-test-slave-7g8g"
server_key: "79b79b1c6da06c22c6a7aaa549ab0a64"
server_key_iv: "d58d4dff10e05cd5d3acf2f60ed1f622"
server_key_version: "encryptionkey@0"
I20260620 23:10:31.484925 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:10:31.488104 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:31.489311 2194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:10:31.489449 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "86898c47bcba4648945fa45651864fe6"
format_stamp: "Formatted at 2026-06-20 23:10:31 on dist-test-slave-7g8g"
server_key: "79b79b1c6da06c22c6a7aaa549ab0a64"
server_key_iv: "d58d4dff10e05cd5d3acf2f60ed1f622"
server_key_version: "encryptionkey@0"
I20260620 23:10:31.489629 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:31.505261 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:31.508760 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:31.508980 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:31.525213 2194 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:45695
I20260620 23:10:31.525244 2260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:45695 every 8 connection(s)
I20260620 23:10:31.526510 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:31.530017 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:31.536160 2261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:10:31.536777 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 2194
I20260620 23:10:31.536949 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:31.537528 12297 external_mini_cluster.cc:1428] Setting key 539db136478a4608ec8d808f6381204e
Jun 20 23:10:31 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:10:31.557235 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.539265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:45695 (local address 127.0.0.1:42712)
0620 23:10:31.540327 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:31.540359 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:31.542041 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:31.542818 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:31.542824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:31.543504 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:31.543836 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:31.543854 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:31.545332 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:31.545335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:31.546378 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:31.546389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:31.547290 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:31.547297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:31.547459 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:31.548031 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:31.548053 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:31.550302 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:31.553245 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:31.553256 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:31.553269 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:31.553649 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:31.554007 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:31.554012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:31.554016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:31.554159 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:31.554861 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:31.554868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:31.555085 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 23:10:31.555307 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 23:10:31.555906 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":130,"thread_start_us":680,"threads_started":1}
I20260620 23:10:31.557250 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.539264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.12.2.126:45695)
0620 23:10:31.540033 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:31.540047 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:31.540476 (+ 429us) server_negotiation.cc:407] Connection header received
0620 23:10:31.542386 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:31.542391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:31.542608 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:31.542788 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:31.544026 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:31.545190 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:31.546506 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:31.547075 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:31.547246 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:31.550518 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:31.550526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:31.550537 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:31.550570 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:31.553131 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:31.553762 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:31.553768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:31.553770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:31.553909 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:31.554252 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:31.554257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:31.554259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:31.554728 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:31.554908 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:31.555430 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 23:10:31.555786 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":554,"thread_start_us":353,"threads_started":1}
WARNING: no policy specified for kudu/127.12.2.125@KRBTEST.COM; defaulting to no policy
I20260620 23:10:31.560196 2261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.125@KRBTEST.COM" created.
I20260620 23:10:31.561269 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.560834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57086)
0620 23:10:31.561005 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:31.561132 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:10:31.561496 2211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
I20260620 23:10:31.562990 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.562604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57102)
0620 23:10:31.562734 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:31.562808 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 23:10:31.563161 2261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:42801: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab.
Entry for principal kudu/127.12.2.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab.
I20260620 23:10:31.588295 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.125:42801
--webserver_interface=127.12.2.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab
--principal=kudu/127.12.2.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:45695,127.12.2.125:42801,127.12.2.124:43979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:10:31.612485 2261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } attempt: 1
I20260620 23:10:31.613636 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.613157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57108)
0620 23:10:31.613356 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:31.613480 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:10:31.614581 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.614223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57120)
0620 23:10:31.614349 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:31.614429 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 23:10:31.614763 2261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:42801: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
I20260620 23:10:31.683014 2261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } attempt: 2
I20260620 23:10:31.684264 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.683713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57134)
0620 23:10:31.683974 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:31.684085 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 23:10:31.685228 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.684841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57140)
0620 23:10:31.684984 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:31.685064 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 23:10:31.685397 2261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:42801: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
I20260620 23:10:31.776654 2261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } attempt: 3
I20260620 23:10:31.777894 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.777374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57152)
0620 23:10:31.777599 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:31.777716 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 23:10:31.779147 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.778719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57164)
0620 23:10:31.778868 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:31.778987 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 23:10:31.779349 2261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:42801: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
W20260620 23:10:31.898193 2268 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:10:31.898514 2268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:31.898586 2268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:31.898664 2268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:31.905774 2268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:31.905870 2268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:31.905936 2268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:31.905985 2268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:31.906042 2268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:31.915858 2268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:45695,127.12.2.125:42801,127.12.2.124:43979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab
--principal=kudu/127.12.2.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.125:42801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.2.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:31.917128 2268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:31.918403 2268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:10:31.922582 2261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } attempt: 4
I20260620 23:10:31.923887 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.923283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57178)
0620 23:10:31.923612 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:31.923724 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260620 23:10:31.925135 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:31.924646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57190)
0620 23:10:31.924862 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:31.924959 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260620 23:10:31.925411 2261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:42801: Network error: Client connection negotiation failed: client connection to 127.12.2.125:42801: connect: Connection refused (error 111)
W20260620 23:10:31.929893 2273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:31.929927 2274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:31.929955 2276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:31.930584 2268 server_base.cc:1034] running on GCE node
I20260620 23:10:31.931337 2268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:31.932631 2268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:31.933846 2268 hybrid_clock.cc:648] HybridClock initialized: now 1781997031933764 us; error 49 us; skew 500 ppm
I20260620 23:10:31.934165 2268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:31 dist-test-slave-7g8g krb5kdc[2181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:31.937119 2268 init.cc:377] Logged in from keytab as kudu/127.12.2.125@KRBTEST.COM (short username kudu)
I20260620 23:10:31.938407 2268 webserver.cc:460] Webserver started at http://127.12.2.125:33147/ using document root <none> and password file <none>
I20260620 23:10:31.939105 2268 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:31.939195 2268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:31.939451 2268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:31.942261 2268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/data/instance:
uuid: "b2d6c3c39405435fac48fbdc8b5d3e26"
format_stamp: "Formatted at 2026-06-20 23:10:31 on dist-test-slave-7g8g"
server_key: "3a9436081ef0545b4175385b9700c6c1"
server_key_iv: "e7c353fec749d83ed0c0616751287377"
server_key_version: "encryptionkey@0"
I20260620 23:10:31.943035 2268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/wal/instance:
uuid: "b2d6c3c39405435fac48fbdc8b5d3e26"
format_stamp: "Formatted at 2026-06-20 23:10:31 on dist-test-slave-7g8g"
server_key: "3a9436081ef0545b4175385b9700c6c1"
server_key_iv: "e7c353fec749d83ed0c0616751287377"
server_key_version: "encryptionkey@0"
I20260620 23:10:31.947913 2268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 23:10:31.951226 2283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:31.952456 2268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:10:31.952633 2268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/wal
uuid: "b2d6c3c39405435fac48fbdc8b5d3e26"
format_stamp: "Formatted at 2026-06-20 23:10:31 on dist-test-slave-7g8g"
server_key: "3a9436081ef0545b4175385b9700c6c1"
server_key_iv: "e7c353fec749d83ed0c0616751287377"
server_key_version: "encryptionkey@0"
I20260620 23:10:31.952824 2268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:31.973907 2268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:31.978062 2268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:31.978319 2268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:31.995184 2268 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.125:42801
I20260620 23:10:31.995203 2334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.125:42801 every 8 connection(s)
I20260620 23:10:31.996516 2268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
I20260620 23:10:32.000144 2335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:32.000391 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 2268
I20260620 23:10:32.000574 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-1/wal/instance
I20260620 23:10:32.001137 12297 external_mini_cluster.cc:1428] Setting key 10be1c2234da7e716b5f1271bd2aeceb
I20260620 23:10:32.008626 2335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.2.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.124@KRBTEST.COM" created.
I20260620 23:10:32.028483 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.011710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.12.2.126:45695)
0620 23:10:32.011919 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.011925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.013007 (+ 1082us) server_negotiation.cc:407] Connection header received
0620 23:10:32.014818 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.014824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.014899 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.015022 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.016468 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.017141 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.018602 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.019074 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.019185 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.022594 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.022599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.022603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.022636 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.025223 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.025929 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.025933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.025937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.026060 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.026597 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.026601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.026604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.026918 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.027053 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.027948 (+ 895us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.028183 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:10:32.029424 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.012085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:45695 (local address 127.0.0.1:42716)
0620 23:10:32.012843 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.012894 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:32.014557 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:32.015240 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:32.015248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:32.015758 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:32.016303 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.016321 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.017288 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.017292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.018460 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.018472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.019283 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.019290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.019515 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.020136 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:32.020157 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:32.022366 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:32.025367 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.025374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.025386 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.025828 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.026219 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.026223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.026226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.026468 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.027085 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:32.027091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:32.027538 (+ 447us) client_negotiation.cc:769] Sending connection context
0620 23:10:32.027834 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 23:10:32.028232 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":323,"threads_started":1}
I20260620 23:10:32.031114 2335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
I20260620 23:10:32.045186 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.031724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57194)
0620 23:10:32.031898 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.031917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:32.032004 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:32.033026 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:32.033032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:32.033459 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:32.033598 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.033607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.035015 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.035021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.035881 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.035893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.036737 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.036744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.036934 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.037528 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:32.037546 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:32.039591 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:32.042654 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.042660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.042665 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.043070 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.043436 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.043442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.043446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.043586 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.044191 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:32.044197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:32.044348 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 23:10:32.044473 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 23:10:32.044756 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:10:32.045447 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.031988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.12.2.125:42801)
0620 23:10:32.032621 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.032626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.032654 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:10:32.032698 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.032703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.032871 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.033018 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.033762 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.034831 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.036020 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.036599 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.036720 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.039818 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.039830 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.039843 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.039879 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.042533 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.043185 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.043196 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.043199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.043325 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.043695 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.043698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.043700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.044053 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.044264 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.044646 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.044856 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":338,"threads_started":1}
I20260620 23:10:32.047829 2335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:10:32.048846 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.048442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34160)
0620 23:10:32.048603 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.048710 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:10:32.049177 2284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
I20260620 23:10:32.050572 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.050223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34168)
0620 23:10:32.050351 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.050432 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:10:32.050820 2335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:43979: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab.
Entry for principal kudu/127.12.2.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab.
I20260620 23:10:32.056427 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.124:43979
--webserver_interface=127.12.2.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab
--principal=kudu/127.12.2.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:45695,127.12.2.125:42801,127.12.2.124:43979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:10:32.100142 2335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } attempt: 1
I20260620 23:10:32.101282 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.100844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34172)
0620 23:10:32.101033 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.101133 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 23:10:32.102190 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.101878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34176)
0620 23:10:32.102000 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.102086 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:10:32.102396 2335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:43979: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
I20260620 23:10:32.170588 2335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } attempt: 2
I20260620 23:10:32.171756 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.171292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34192)
0620 23:10:32.171473 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.171583 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 23:10:32.172775 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.172467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34204)
0620 23:10:32.172597 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.172665 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 23:10:32.172959 2335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:43979: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
I20260620 23:10:32.224676 2261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } attempt: 5
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
I20260620 23:10:32.237391 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.225483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.12.2.125:42801)
0620 23:10:32.225679 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.225685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.225778 (+ 93us) server_negotiation.cc:407] Connection header received
0620 23:10:32.225965 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.225970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.226060 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.226144 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.226979 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.227694 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.228829 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.229240 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.229346 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.232371 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.232376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.232379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.232411 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.234997 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.235613 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.235618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.235621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.235740 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.236071 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.236076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.236080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.236379 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.236509 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.236872 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.237086 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:10:32.237397 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.225452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57200)
0620 23:10:32.225678 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.225699 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:32.225840 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:32.226202 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:32.226206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:32.226615 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:32.226798 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.226809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.227830 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.227835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.228685 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.228696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.229390 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.229396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.229594 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.230238 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:32.230255 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:32.232204 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:32.235125 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.235131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.235135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.235496 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.235840 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.235845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.235848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.235974 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.236489 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:32.236493 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:32.236645 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 23:10:32.236759 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:10:32.237032 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 23:10:32.238448 2261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:10:32.239609 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.239216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34210)
0620 23:10:32.239370 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.239443 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:10:32.240482 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.240184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34226)
0620 23:10:32.240316 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.240394 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 23:10:32.240670 2261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:43979: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
I20260620 23:10:32.264173 2335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } attempt: 3
I20260620 23:10:32.265326 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.264841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34232)
0620 23:10:32.265064 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.265159 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 23:10:32.266445 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.265996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34242)
0620 23:10:32.266212 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.266291 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260620 23:10:32.266659 2335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:43979: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
I20260620 23:10:32.291872 2261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } attempt: 1
I20260620 23:10:32.292949 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.292513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34246)
0620 23:10:32.292698 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.292785 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 23:10:32.294030 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.293699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34248)
0620 23:10:32.293839 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.293909 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 23:10:32.294234 2261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:43979: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
I20260620 23:10:32.362524 2261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } attempt: 2
I20260620 23:10:32.363618 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.363172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34252)
0620 23:10:32.363366 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.363462 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 23:10:32.364578 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.364266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34262)
0620 23:10:32.364401 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.364477 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:10:32.364768 2261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:43979: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
W20260620 23:10:32.373400 2342 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:10:32.373740 2342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:32.373835 2342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:32.373942 2342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:32.381322 2342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:32.381428 2342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:32.381500 2342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:32.381570 2342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:32.381654 2342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:32.391564 2342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:45695,127.12.2.125:42801,127.12.2.124:43979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab
--principal=kudu/127.12.2.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.124:43979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.2.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:32.392886 2342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:32.394223 2342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:32.404903 2347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:32.404953 2348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:32.404979 2350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:32.405694 2342 server_base.cc:1034] running on GCE node
I20260620 23:10:32.406365 2342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:32.407637 2342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:32.408835 2342 hybrid_clock.cc:648] HybridClock initialized: now 1781997032408775 us; error 36 us; skew 500 ppm
I20260620 23:10:32.409135 2342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:10:32.409842 2335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } attempt: 4
I20260620 23:10:32.410776 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.410343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34276)
0620 23:10:32.410551 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.410632 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997032, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:32.411801 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.411440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34278)
0620 23:10:32.411591 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.411673 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:10:32.412030 2335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:43979: Network error: Client connection negotiation failed: client connection to 127.12.2.124:43979: connect: Connection refused (error 111)
I20260620 23:10:32.412336 2342 init.cc:377] Logged in from keytab as kudu/127.12.2.124@KRBTEST.COM (short username kudu)
I20260620 23:10:32.413728 2342 webserver.cc:460] Webserver started at http://127.12.2.124:45369/ using document root <none> and password file <none>
I20260620 23:10:32.414361 2342 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:32.414458 2342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:32.414748 2342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:32.417634 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/data/instance:
uuid: "c9cfb4b507c84eb9b682b5504893e97b"
format_stamp: "Formatted at 2026-06-20 23:10:32 on dist-test-slave-7g8g"
server_key: "9495116bcc6e9bfddd37fbc7d17f1483"
server_key_iv: "72352195cc71a3dbd4be420d3db50323"
server_key_version: "encryptionkey@0"
I20260620 23:10:32.418355 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/wal/instance:
uuid: "c9cfb4b507c84eb9b682b5504893e97b"
format_stamp: "Formatted at 2026-06-20 23:10:32 on dist-test-slave-7g8g"
server_key: "9495116bcc6e9bfddd37fbc7d17f1483"
server_key_iv: "72352195cc71a3dbd4be420d3db50323"
server_key_version: "encryptionkey@0"
I20260620 23:10:32.423327 2342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 23:10:32.426596 2357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:32.427887 2342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:32.428040 2342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/wal
uuid: "c9cfb4b507c84eb9b682b5504893e97b"
format_stamp: "Formatted at 2026-06-20 23:10:32 on dist-test-slave-7g8g"
server_key: "9495116bcc6e9bfddd37fbc7d17f1483"
server_key_iv: "72352195cc71a3dbd4be420d3db50323"
server_key_version: "encryptionkey@0"
I20260620 23:10:32.428229 2342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:32.449026 2342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:32.453264 2342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:32.453513 2342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:32.469660 2342 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.124:43979
I20260620 23:10:32.469678 2408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.124:43979 every 8 connection(s)
I20260620 23:10:32.470953 2342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
I20260620 23:10:32.470970 2261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } attempt: 3
I20260620 23:10:32.475384 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:32.478559 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 2342
I20260620 23:10:32.478747 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996696601630-12297-0/minicluster-data/master-2/wal/instance
I20260620 23:10:32.479310 12297 external_mini_cluster.cc:1428] Setting key bebf3b41e644b1d7f71dd1edfb553ea9
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
I20260620 23:10:32.486366 2409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:10:32.494550 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.471593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34288)
0620 23:10:32.471867 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.471886 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:32.472032 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:32.476200 (+ 4168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:32.476205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:32.476723 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:32.476920 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.476933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.478526 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.478530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.479683 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.479697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.481291 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.481298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.481509 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.482221 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:32.482247 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:32.485302 (+ 3055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:32.490921 (+ 5619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.490943 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.490948 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.491503 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.492101 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.492107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.492111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.492278 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.493446 (+ 1168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:32.493452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:32.493664 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 23:10:32.493803 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:10:32.494118 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 23:10:32.497381 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.472792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.12.2.124:43979)
0620 23:10:32.473753 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.473767 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.473807 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:10:32.475432 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.475452 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.475842 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.476168 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.477078 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.478342 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.479815 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.481039 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.485522 (+ 4483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.485911 (+ 389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.485922 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.485944 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.485993 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.490755 (+ 4762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.491685 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.491693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.491703 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.491918 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.492414 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.492419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.492422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.493297 (+ 875us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.495114 (+ 1817us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.495386 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.495872 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":591,"threads_started":1}
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997032, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:10:32.504783 2261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6: Bootstrap starting.
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
I20260620 23:10:32.508388 2261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:32.509335 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.489834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.12.2.126:45695)
0620 23:10:32.490045 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.490052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.493668 (+ 3616us) server_negotiation.cc:407] Connection header received
0620 23:10:32.494552 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.494557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.494631 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.494697 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.496213 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.496892 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.498799 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.499310 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.499417 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.502794 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.502801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.502806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.502841 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.506184 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.507066 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.507071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.507073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.507245 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.507750 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.507756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.507759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.508111 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.508253 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.508767 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.508979 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:10:32.509382 2414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.489779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:45695 (local address 127.0.0.1:42722)
0620 23:10:32.493535 (+ 3756us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.493583 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:32.494407 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:32.494920 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:32.494946 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:32.495831 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:32.496062 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.496071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.497042 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.497048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.498654 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.498668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.499484 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.499491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.499655 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.500252 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:32.500269 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:32.502590 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:32.506288 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.506299 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.506306 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.506900 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.507394 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.507399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.507403 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.507612 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.508233 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:32.508241 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:32.508432 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 23:10:32.508645 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 23:10:32.508960 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3511,"thread_start_us":3371,"threads_started":1}
I20260620 23:10:32.509683 2261 log.cc:824] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:32.511279 2409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:10:32.512655 2413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.480266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.12.2.124:43979)
0620 23:10:32.495041 (+ 14775us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.495046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.495065 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:10:32.495106 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.495110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.495174 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.495281 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.496964 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.498195 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.499895 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.502257 (+ 2362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.502383 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.506299 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.506306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.506311 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.506401 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.509541 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.510162 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.510167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.510170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.510296 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.510743 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.510748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.510751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.511116 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.511409 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.512020 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.512225 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14620,"spinlock_wait_cycles":19840,"thread_start_us":14446,"threads_started":1}
I20260620 23:10:32.513062 2261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6: No bootstrap required, opened a new log
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997032, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
I20260620 23:10:32.522879 2261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } }
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:10:32.523257 2261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86898c47bcba4648945fa45651864fe6, State: Initialized, Role: FOLLOWER
I20260620 23:10:32.523766 2261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } }
I20260620 23:10:32.524714 2416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } } }
I20260620 23:10:32.525506 2416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:32.526429 2261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:32.532706 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.511965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.12.2.125:42801)
0620 23:10:32.512142 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.512148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.512165 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:10:32.512225 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.512229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.512306 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.512391 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.513179 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.514042 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.515514 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.516137 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.516278 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.519381 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.519388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.519394 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.519436 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.522682 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.523471 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.523476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.523479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.523645 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.524079 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.524084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.524087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.524441 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.530516 (+ 6075us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.532050 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.532296 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:10:32.533216 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.515009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.12.2.126:45695)
0620 23:10:32.515164 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.515169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.515243 (+ 74us) server_negotiation.cc:407] Connection header received
0620 23:10:32.515444 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.515449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.515533 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.515632 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.516985 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.518148 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.519700 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.520234 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.520357 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.524411 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.524418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.524422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.524461 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.529958 (+ 5497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.530819 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.530825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.530829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.531039 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.531543 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.531547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.531550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.531911 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.532162 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.532695 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.532900 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:10:32.533603 2414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.511859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:42801 (local address 127.0.0.1:57214)
0620 23:10:32.511994 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.512010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:32.512098 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:32.512423 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:32.512427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:32.512875 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:32.513024 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.513032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.514215 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.514220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.515386 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.515400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.516320 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.516327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.516530 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.517104 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:32.517121 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:32.519186 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:32.522813 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.522819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.522823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.523370 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.523764 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.523769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.523773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.523937 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.531549 (+ 7612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:32.531555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:32.531747 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 23:10:32.532942 (+ 1195us) client_negotiation.cc:240] Negotiation successful
0620 23:10:32.533259 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 23:10:32.534637 2409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997032, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
I20260620 23:10:32.550593 2414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.535453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34318)
0620 23:10:32.535773 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.535792 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:32.535895 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:32.536698 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:32.536703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:32.537219 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:32.537415 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.537427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.538834 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.538839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.540282 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.540296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.541227 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.541234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.541435 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.541974 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:32.541991 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:32.544615 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:32.548091 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.548096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.548099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.548559 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.548918 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.548923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.548926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.549065 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.549682 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:32.549687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:32.549842 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:10:32.549971 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 23:10:32.550240 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260620 23:10:32.551627 2413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.536068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.12.2.124:43979)
0620 23:10:32.536270 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.536276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.536293 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:10:32.536339 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.536344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.536413 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.536500 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.537572 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.538695 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.540438 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.541048 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.541171 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.544800 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.544806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.544811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.544845 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.547967 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.548682 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.548686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.548690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.548815 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.549183 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.549187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.549190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.549563 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.551001 (+ 1438us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.551119 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.551323 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:10:32.553090 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.536234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.12.2.125:42801)
0620 23:10:32.536385 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.536390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.536625 (+ 235us) server_negotiation.cc:407] Connection header received
0620 23:10:32.536811 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.536817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.536882 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.536953 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.538620 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.539751 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.541442 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.542047 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.542182 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.547046 (+ 4864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.547053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.547057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.547095 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.550205 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.551016 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.551022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.551025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.551176 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.551600 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.551605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.551608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.551953 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.552126 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.552578 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.552797 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:10:32.557855 2409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b: Bootstrap starting.
I20260620 23:10:32.559082 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 23:10:32.559217 2427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:10:32.559347 2427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:10:32.561787 2409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:32.546967 2194 master_runner.cc:186] Error getting master registration for 127.12.2.125:42801: OK, instance_id { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" instance_seqno: 1781997031990358 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:10:32.562439 2194 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:10:32.562470 2236 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "86898c47bcba4648945fa45651864fe6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42728
I20260620 23:10:32.562837 2236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:10:32.563042 2236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:32.563683 2409 log.cc:824] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:32.566329 2236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } }
I20260620 23:10:32.568270 2236 leader_election.cc:288] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [CANDIDATE]: Term 1 election: Requested vote from peers b2d6c3c39405435fac48fbdc8b5d3e26 (127.12.2.125:42801), c9cfb4b507c84eb9b682b5504893e97b (127.12.2.124:43979)
I20260620 23:10:32.569295 2409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b: No bootstrap required, opened a new log
I20260620 23:10:32.579877 2409 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } }
I20260620 23:10:32.580313 2409 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cfb4b507c84eb9b682b5504893e97b, State: Initialized, Role: FOLLOWER
I20260620 23:10:32.580832 2409 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } }
I20260620 23:10:32.582049 2433 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } } }
I20260620 23:10:32.582981 2433 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:32.583206 2409 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:32.583287 2384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86898c47bcba4648945fa45651864fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c9cfb4b507c84eb9b682b5504893e97b"
I20260620 23:10:32.583719 2384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:32.587251 2384 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86898c47bcba4648945fa45651864fe6 in term 1.
I20260620 23:10:32.588049 2210 leader_election.cc:302] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86898c47bcba4648945fa45651864fe6, c9cfb4b507c84eb9b682b5504893e97b; no voters:
I20260620 23:10:32.588708 2416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:32.595245 2416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [term 1 LEADER]: Becoming Leader. State: Replica: 86898c47bcba4648945fa45651864fe6, State: Running, Role: LEADER
I20260620 23:10:32.596359 2416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } }
I20260620 23:10:32.610535 2436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86898c47bcba4648945fa45651864fe6. Latest consensus state: current_term: 1 leader_uuid: "86898c47bcba4648945fa45651864fe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } } }
I20260620 23:10:32.611145 2436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:32.626408 2442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:32.630203 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.594589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.12.2.126:45695)
0620 23:10:32.594764 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.594770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.605473 (+ 10703us) server_negotiation.cc:407] Connection header received
0620 23:10:32.606912 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.606920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.607023 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.607105 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.611304 (+ 4199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.612504 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.616020 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.616730 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.616826 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.622581 (+ 5755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.622590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.622596 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.622636 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.626953 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.627710 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.627716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.627719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.627871 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.628338 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.628343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.628346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.628730 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.628948 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.629529 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.629750 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:10:32.636793 2442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:32.640537 2413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.601441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.12.2.124:43979)
0620 23:10:32.601630 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.601636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.617162 (+ 15526us) server_negotiation.cc:407] Connection header received
0620 23:10:32.617631 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.617636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.617717 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.617787 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.622651 (+ 4864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.623761 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.627021 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.627688 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.627776 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.633271 (+ 5495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.633278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.633283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.633323 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.636684 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.638152 (+ 1468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.638158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.638161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.638301 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.638719 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.638725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.638728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.639178 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.639416 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.639930 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.640176 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:10:32.645409 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.606011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.12.2.125:42801)
0620 23:10:32.606206 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.606212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.622468 (+ 16256us) server_negotiation.cc:407] Connection header received
0620 23:10:32.622751 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.622756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.622836 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.622926 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.628751 (+ 5825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.629758 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.633772 (+ 4014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.634636 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.634737 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.638242 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.638249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.638253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.638294 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.641557 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.643303 (+ 1746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.643308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.643312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.643471 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.643912 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.643917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.643920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.644295 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.644550 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.644847 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.645063 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:10:32.654086 2384 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [term 1 FOLLOWER]: Refusing update from remote peer 86898c47bcba4648945fa45651864fe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:32.655244 2436 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:32.657150 2452 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:10:32.662835 2342 master_runner.cc:186] Error getting master registration for 127.12.2.125:42801: OK, instance_id { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" instance_seqno: 1781997031990358 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:10:32.673802 2342 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:10:32.678092 2433 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86898c47bcba4648945fa45651864fe6. Latest consensus state: current_term: 1 leader_uuid: "86898c47bcba4648945fa45651864fe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } } }
I20260620 23:10:32.678376 2433 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:32.680014 2416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86898c47bcba4648945fa45651864fe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } } }
I20260620 23:10:32.680406 2416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:32.680858 2453 mvcc.cc:205] Tried to move back new op lower bound from 7299059845741465600 to 7299059845524930560. Current Snapshot: MvccSnapshot[applied={T|T < 7299059845741465600}]
I20260620 23:10:32.682886 2442 catalog_manager.cc:1347] Generated new cluster ID: f181984f642348caad068a5446bcd01b
I20260620 23:10:32.683019 2442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:32.684068 2433 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86898c47bcba4648945fa45651864fe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } } }
I20260620 23:10:32.684319 2433 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:32.711211 2335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } attempt: 5
Jun 20 23:10:32 dist-test-slave-7g8g krb5kdc[2181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997031, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
I20260620 23:10:32.731916 2413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.712433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.12.2.124:43979)
0620 23:10:32.712615 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.712620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.712638 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:10:32.712688 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.712693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.712764 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.712860 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.713809 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.714969 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.716713 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.717174 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.717285 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.721149 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:32.721156 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:32.721161 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:32.721205 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:32.724502 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.727970 (+ 3468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.727993 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.727997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.728152 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:32.728566 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:32.728571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:32.728574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:32.729003 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:32.729150 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.731224 (+ 2074us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.731550 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:10:32.732450 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.711907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:43979 (local address 127.0.0.1:34326)
0620 23:10:32.712093 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:32.712115 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:32.712224 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:32.712893 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:32.712898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:32.713430 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:32.713636 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.713649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.715118 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.715124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.716562 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:32.716577 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.717327 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.717333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:32.717559 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.718168 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:32.718191 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:32.720962 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:32.726423 (+ 5461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.726428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.726432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.727846 (+ 1414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.728269 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:32.728274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:32.728277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:32.728454 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:32.729215 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:32.729220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:32.729384 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 23:10:32.729539 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 23:10:32.729840 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:10:32.732710 2442 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:32.734561 2442 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:32.738063 2335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26: Bootstrap starting.
I20260620 23:10:32.742038 2335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:32.744491 2335 log.cc:824] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:32.747221 2442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6: Generated new TSK 0
I20260620 23:10:32.748208 2442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:32.757066 2335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26: No bootstrap required, opened a new log
I20260620 23:10:32.779210 2335 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } }
I20260620 23:10:32.779712 2335 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d6c3c39405435fac48fbdc8b5d3e26, State: Initialized, Role: FOLLOWER
I20260620 23:10:32.780331 2335 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } }
I20260620 23:10:32.781836 2460 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } } }
I20260620 23:10:32.782599 2460 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:32.783111 2335 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:32.783174 2310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86898c47bcba4648945fa45651864fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26"
I20260620 23:10:32.783644 2310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:32.787959 2310 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86898c47bcba4648945fa45651864fe6 in term 1.
I20260620 23:10:32.788844 2309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [term 1 FOLLOWER]: Refusing update from remote peer 86898c47bcba4648945fa45651864fe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:32.790884 2457 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:32.827958 2460 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86898c47bcba4648945fa45651864fe6. Latest consensus state: current_term: 1 leader_uuid: "86898c47bcba4648945fa45651864fe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } } }
I20260620 23:10:32.828323 2460 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:32.828718 2469 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86898c47bcba4648945fa45651864fe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } } }
I20260620 23:10:32.829191 2469 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:32.841769 2479 catalog_manager.cc:1259] Loaded cluster ID: f181984f642348caad068a5446bcd01b
I20260620 23:10:32.841974 2479 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26: loading cluster ID for follower catalog manager: success
I20260620 23:10:32.846086 2479 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26: acquiring CA information for follower catalog manager: success
I20260620 23:10:32.848783 2479 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2d6c3c39405435fac48fbdc8b5d3e26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:10:32.865046 2263 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:32.851854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.12.2.126:45695)
0620 23:10:32.852086 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.852092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.862771 (+ 10679us) server_negotiation.cc:407] Connection header received
0620 23:10:32.863369 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.863379 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.863465 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.863546 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.864790 (+ 1244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42750: BlockingRecv error: recv got EOF from 127.0.0.1:42750 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 23:10:32.868639 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:32.851585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.12.2.124:43979)
0620 23:10:32.851759 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.851766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.865761 (+ 13995us) server_negotiation.cc:407] Connection header received
0620 23:10:32.866707 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.866712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.866791 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.866857 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.868383 (+ 1526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34342: BlockingRecv error: recv got EOF from 127.0.0.1:34342 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 23:10:32.869982 2339 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:32.865947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.12.2.125:42801)
0620 23:10:32.866126 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.866132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.866364 (+ 232us) server_negotiation.cc:407] Connection header received
0620 23:10:32.867077 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.867083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.867184 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.867252 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.869753 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57240: BlockingRecv error: recv got EOF from 127.0.0.1:57240 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:10:32.886317 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.875886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.12.2.125:42801)
0620 23:10:32.876063 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.876069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.879427 (+ 3358us) server_negotiation.cc:407] Connection header received
0620 23:10:32.879663 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.879669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.879881 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.880049 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:32.880499 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.881666 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.883545 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.884193 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.884337 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.884899 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:32.885654 (+ 755us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:32.885794 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.885884 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.885978 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 23:10:32.909127 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:32.900922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.12.2.124:43979)
0620 23:10:32.901115 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.901121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.907547 (+ 6426us) server_negotiation.cc:407] Connection header received
0620 23:10:32.907778 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.907785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.907940 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.908057 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.908901 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34352: BlockingRecv error: recv got EOF from 127.0.0.1:34352 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:10:32.920498 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.913872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.12.2.125:42801)
0620 23:10:32.914060 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.914066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.914542 (+ 476us) server_negotiation.cc:407] Connection header received
0620 23:10:32.914727 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.914733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.914838 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.914963 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:32.915407 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.916561 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.918480 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.919168 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.919292 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.919729 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:32.919880 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:32.919980 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.920099 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.920181 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 23:10:32.935540 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:32.933361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.12.2.124:43979)
0620 23:10:32.933535 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.933541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.934042 (+ 501us) server_negotiation.cc:407] Connection header received
0620 23:10:32.934253 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.934259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.934348 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.934455 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.935289 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34356: BlockingRecv error: recv got EOF from 127.0.0.1:34356 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:10:32.947881 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.941297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.12.2.125:42801)
0620 23:10:32.941463 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.941469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.941990 (+ 521us) server_negotiation.cc:407] Connection header received
0620 23:10:32.942165 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.942171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.942252 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.942342 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:32.942779 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.943973 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.945814 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.946509 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.946634 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.947094 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:32.947228 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:32.947315 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.947473 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.947555 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 23:10:32.956043 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:32.954288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34366 (local address 127.12.2.124:43979)
0620 23:10:32.954463 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.954469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.954488 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:10:32.954712 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.954717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.954804 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.954920 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.955800 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34366: BlockingRecv error: recv got EOF from 127.0.0.1:34366 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:10:32.975497 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:32.968732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.12.2.125:42801)
0620 23:10:32.968906 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.968912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.969405 (+ 493us) server_negotiation.cc:407] Connection header received
0620 23:10:32.969602 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.969607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.969699 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.969810 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:32.970239 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.971438 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.973352 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:32.974077 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:32.974205 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:32.974697 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:32.974836 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:32.974983 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:32.975113 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 23:10:32.975200 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 23:10:32.993017 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:32.984264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34368 (local address 127.12.2.124:43979)
0620 23:10:32.984525 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:32.984531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:32.991465 (+ 6934us) server_negotiation.cc:407] Connection header received
0620 23:10:32.991671 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:32.991677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:32.991775 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:32.991876 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:32.992778 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34368: BlockingRecv error: recv got EOF from 127.0.0.1:34368 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 23:10:33.034267 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:33.027165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.12.2.125:42801)
0620 23:10:33.027359 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.027364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.027918 (+ 554us) server_negotiation.cc:407] Connection header received
0620 23:10:33.028209 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.028215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.028314 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.028412 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:33.028902 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.030050 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.032046 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.032849 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.032961 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:33.033423 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:33.033610 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:33.033695 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:33.033854 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 23:10:33.033933 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 23:10:33.043334 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:33.041217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34372 (local address 127.12.2.124:43979)
0620 23:10:33.041380 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.041385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.041832 (+ 447us) server_negotiation.cc:407] Connection header received
0620 23:10:33.041967 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.041972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.042063 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.042160 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:33.043104 (+ 944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34372: BlockingRecv error: recv got EOF from 127.0.0.1:34372 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:10:33.069193 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:33.062635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.12.2.125:42801)
0620 23:10:33.062816 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.062823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.063353 (+ 530us) server_negotiation.cc:407] Connection header received
0620 23:10:33.063559 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.063568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.063656 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.063758 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:33.064212 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.065306 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.067221 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.067832 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.067951 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:33.068424 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:33.068595 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:33.068685 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:33.068811 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 23:10:33.068908 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 23:10:33.079371 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:33.075800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.12.2.124:43979)
0620 23:10:33.075982 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.075987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.077988 (+ 2001us) server_negotiation.cc:407] Connection header received
0620 23:10:33.078139 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.078144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.078230 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.078322 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:33.079147 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34388: BlockingRecv error: recv got EOF from 127.0.0.1:34388 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:10:33.120065 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:33.114907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.12.2.125:42801)
0620 23:10:33.115162 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.115168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.115470 (+ 302us) server_negotiation.cc:407] Connection header received
0620 23:10:33.115632 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.115638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.115744 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.115851 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:33.116243 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.116996 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.118329 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.118829 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.118964 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:33.119364 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:33.119471 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:33.119556 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:33.119704 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 23:10:33.119786 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260620 23:10:33.128343 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:33.126230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.12.2.124:43979)
0620 23:10:33.126431 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.126437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.126986 (+ 549us) server_negotiation.cc:407] Connection header received
0620 23:10:33.127176 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.127182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.127294 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.127403 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:33.128124 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34394: BlockingRecv error: recv got EOF from 127.0.0.1:34394 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:10:33.205744 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:33.199846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.12.2.125:42801)
0620 23:10:33.200076 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.200082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.200443 (+ 361us) server_negotiation.cc:407] Connection header received
0620 23:10:33.200647 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.200651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.200720 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.200796 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:33.201263 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.202081 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.204027 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.204495 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.204593 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:33.205048 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:33.205165 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:33.205231 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:33.205380 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:10:33.205474 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 23:10:33.212915 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:33.211176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.12.2.124:43979)
0620 23:10:33.211364 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.211370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.211650 (+ 280us) server_negotiation.cc:407] Connection header received
0620 23:10:33.211802 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.211805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.211875 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.211964 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:33.212696 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34406: BlockingRecv error: recv got EOF from 127.0.0.1:34406 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:10:33.266880 2457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86898c47bcba4648945fa45651864fe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86898c47bcba4648945fa45651864fe6" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 45695 } } peers { permanent_uuid: "b2d6c3c39405435fac48fbdc8b5d3e26" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 42801 } } peers { permanent_uuid: "c9cfb4b507c84eb9b682b5504893e97b" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 43979 } } }
I20260620 23:10:33.267189 2457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86898c47bcba4648945fa45651864fe6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:33.349817 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:33.344896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.12.2.125:42801)
0620 23:10:33.345104 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.345110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.345448 (+ 338us) server_negotiation.cc:407] Connection header received
0620 23:10:33.345658 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.345662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.345730 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.345805 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:33.346235 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.346925 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.348275 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.348668 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.348773 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:33.349148 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:33.349252 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:33.349325 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:33.349465 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 23:10:33.349547 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 23:10:33.356976 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:33.355136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.12.2.124:43979)
0620 23:10:33.355309 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.355314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.355771 (+ 457us) server_negotiation.cc:407] Connection header received
0620 23:10:33.355892 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.355896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.355969 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.356044 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:33.356756 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34422: BlockingRecv error: recv got EOF from 127.0.0.1:34422 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:10:33.622398 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:33.617534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.12.2.125:42801)
0620 23:10:33.617766 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.617772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.617970 (+ 198us) server_negotiation.cc:407] Connection header received
0620 23:10:33.618179 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.618183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.618263 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.618343 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:33.618765 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.619487 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.620841 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:33.621259 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:33.621368 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:33.621752 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:33.621858 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:33.621931 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:33.622084 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 23:10:33.622165 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260620 23:10:33.629444 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:33.627816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.12.2.124:43979)
0620 23:10:33.628020 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:33.628026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:33.628201 (+ 175us) server_negotiation.cc:407] Connection header received
0620 23:10:33.628325 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:33.628329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:33.628393 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:33.628471 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:33.629209 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34426: BlockingRecv error: recv got EOF from 127.0.0.1:34426 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:10:33.661180 2452 catalog_manager.cc:1259] Loaded cluster ID: f181984f642348caad068a5446bcd01b
I20260620 23:10:33.661305 2452 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b: loading cluster ID for follower catalog manager: success
I20260620 23:10:33.663883 2452 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b: acquiring CA information for follower catalog manager: success
I20260620 23:10:33.665691 2452 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9cfb4b507c84eb9b682b5504893e97b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:10:34.149031 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:34.144413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.12.2.125:42801)
0620 23:10:34.145047 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:34.145052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:34.145069 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:10:34.145132 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:34.145136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:34.145198 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:34.145282 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:34.145641 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:34.146350 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:34.147628 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:34.148014 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:34.148110 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:34.148410 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:34.148504 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:34.148562 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:34.148663 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 23:10:34.148722 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":433,"threads_started":1}
I20260620 23:10:34.159293 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:34.154113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.12.2.124:43979)
0620 23:10:34.154543 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:34.154547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:34.154625 (+ 78us) server_negotiation.cc:407] Connection header received
0620 23:10:34.154761 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:34.154765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:34.154908 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:34.155043 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:34.155351 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:34.156069 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:34.157305 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:34.157724 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:34.157812 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:34.158254 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:34.158742 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:34.158847 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:34.158961 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 23:10:34.159029 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":224,"threads_started":1}
I20260620 23:10:34.172952 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:34.165044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.12.2.126:45695)
0620 23:10:34.166294 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:34.166301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:34.166412 (+ 111us) server_negotiation.cc:407] Connection header received
0620 23:10:34.166551 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:34.166557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:34.166777 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:34.166966 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:34.167863 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:34.168862 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:34.170511 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:34.171141 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:34.171266 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:34.171792 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:34.172114 (+ 322us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:34.172217 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:34.172622 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 23:10:34.172712 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":1016,"threads_started":1}
I20260620 23:10:34.173501 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:34.165744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.12.2.125:42801)
0620 23:10:34.165927 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:34.165934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:34.167500 (+ 1566us) server_negotiation.cc:407] Connection header received
0620 23:10:34.167784 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:34.167789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:34.167874 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:34.167936 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:34.168777 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:34.169801 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:34.171577 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:34.172202 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:34.172326 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:34.172793 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:34.172922 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:34.173005 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:34.173180 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 23:10:34.173260 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:10:34.174269 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:34.165081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.12.2.124:43979)
0620 23:10:34.165930 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:34.165934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:34.167280 (+ 1346us) server_negotiation.cc:407] Connection header received
0620 23:10:34.167763 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:34.167769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:34.167900 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:34.167987 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:34.169242 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:34.170446 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:34.172393 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:34.172909 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:34.173000 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:34.173534 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:34.173649 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:34.173728 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:34.173919 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 23:10:34.173998 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725}
I20260620 23:10:34.177721 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 2194
I20260620 23:10:34.179109 2256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:34.286466 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 2194
I20260620 23:10:34.302815 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 2268
I20260620 23:10:34.303913 2330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:34.409920 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 2268
I20260620 23:10:34.425613 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 2342
I20260620 23:10:34.426638 2404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:34.530731 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 2342
2026-06-20T23:10:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4679 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:34 dist-test-slave-7g8g krb5kdc[2615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:34 dist-test-slave-7g8g krb5kdc[2615](info): set up 2 sockets
Jun 20 23:10:34 dist-test-slave-7g8g krb5kdc[2615](info): commencing operation
krb5kdc: starting...
W20260620 23:10:35.551198 12297 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/test-user.keytab.
Jun 20 23:10:35 dist-test-slave-7g8g krb5kdc[2615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.2.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
Entry for principal kudu/127.12.2.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab.
I20260620 23:10:35.792016 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:33375
--webserver_interface=127.12.2.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:33375,127.12.2.125:45251,127.12.2.124:34017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:36.110114 2628 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:10:36.110437 2628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:36.110538 2628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:36.110651 2628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:36.117805 2628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:36.117895 2628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:36.117975 2628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:36.118041 2628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:36.118124 2628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:36.127665 2628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:33375,127.12.2.125:45251,127.12.2.124:34017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.126.keytab
--principal=kudu/127.12.2.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.126:33375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.2.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:36.128898 2628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:36.130162 2628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:36.140960 2634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:36.140995 2636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:36.141029 2633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:36.141722 2628 server_base.cc:1034] running on GCE node
I20260620 23:10:36.142460 2628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:36.143805 2628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:36.145021 2628 hybrid_clock.cc:648] HybridClock initialized: now 1781997036144956 us; error 38 us; skew 500 ppm
I20260620 23:10:36.145323 2628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:36 dist-test-slave-7g8g krb5kdc[2615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997036, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:36.148273 2628 init.cc:377] Logged in from keytab as kudu/127.12.2.126@KRBTEST.COM (short username kudu)
I20260620 23:10:36.149753 2628 webserver.cc:460] Webserver started at http://127.12.2.126:43033/ using document root <none> and password file <none>
I20260620 23:10:36.150417 2628 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:36.150527 2628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:36.150816 2628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:36.153766 2628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/data/instance:
uuid: "06d97ee62bf8451fbf2d91150f01f159"
format_stamp: "Formatted at 2026-06-20 23:10:36 on dist-test-slave-7g8g"
server_key: "1ecb51123bb83d065c41a94582eafc95"
server_key_iv: "4542e189b7c9c42a1572674488431fc2"
server_key_version: "encryptionkey@0"
I20260620 23:10:36.154520 2628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/wal/instance:
uuid: "06d97ee62bf8451fbf2d91150f01f159"
format_stamp: "Formatted at 2026-06-20 23:10:36 on dist-test-slave-7g8g"
server_key: "1ecb51123bb83d065c41a94582eafc95"
server_key_iv: "4542e189b7c9c42a1572674488431fc2"
server_key_version: "encryptionkey@0"
I20260620 23:10:36.159571 2628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:10:36.163074 2643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:36.164431 2628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:10:36.164614 2628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/wal
uuid: "06d97ee62bf8451fbf2d91150f01f159"
format_stamp: "Formatted at 2026-06-20 23:10:36 on dist-test-slave-7g8g"
server_key: "1ecb51123bb83d065c41a94582eafc95"
server_key_iv: "4542e189b7c9c42a1572674488431fc2"
server_key_version: "encryptionkey@0"
I20260620 23:10:36.164820 2628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:36.184098 2628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:36.188246 2628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:36.188505 2628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:36.205317 2628 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.126:33375
I20260620 23:10:36.205338 2694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.126:33375 every 8 connection(s)
I20260620 23:10:36.206693 2628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:36.207680 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 2628
I20260620 23:10:36.207868 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-0/wal/instance
I20260620 23:10:36.208393 12297 external_mini_cluster.cc:1428] Setting key 34e17b381192172c766b836fa8c0d6bf
I20260620 23:10:36.211406 2695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:36.219676 2695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:10:36 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997036, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.2.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.125@KRBTEST.COM" created.
I20260620 23:10:36.240875 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.222830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:33375 (local address 127.0.0.1:60662)
0620 23:10:36.223919 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.223959 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:36.225619 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:36.226401 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:36.226408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:36.227120 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:36.227443 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:36.227460 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.228948 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.228951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:36.230054 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:36.230066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.230914 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.230922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:36.231110 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:36.231702 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:36.231724 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:36.233715 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:36.237243 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:36.237253 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:36.237262 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:36.237619 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:36.238060 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:36.238063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:36.238066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:36.238214 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:36.239067 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:36.239073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:36.239254 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 23:10:36.239446 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 23:10:36.239825 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":678,"threads_started":1}
I20260620 23:10:36.240904 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.222829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.12.2.126:33375)
0620 23:10:36.223647 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:36.223662 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:36.224074 (+ 412us) server_negotiation.cc:407] Connection header received
0620 23:10:36.225950 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:36.225955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:36.226194 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:36.226358 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:36.227640 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.228805 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.230172 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.230701 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.230862 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:36.233912 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:36.233919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:36.233930 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:36.233957 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:36.237104 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:36.237763 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:36.237771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:36.237774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:36.237953 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:36.238321 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:36.238326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:36.238329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:36.238911 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:36.239133 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:36.239626 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 23:10:36.239985 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":598,"thread_start_us":352,"threads_started":1}
I20260620 23:10:36.243717 2695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:10:36.244796 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.244399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60854)
0620 23:10:36.244596 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.244695 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 23:10:36.245092 2645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
I20260620 23:10:36.246573 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.246191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60862)
0620 23:10:36.246350 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.246427 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:10:36.246771 2695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:45251: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab.
Entry for principal kudu/127.12.2.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab.
I20260620 23:10:36.266307 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.125:45251
--webserver_interface=127.12.2.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab
--principal=kudu/127.12.2.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:33375,127.12.2.125:45251,127.12.2.124:34017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:10:36.296094 2695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } attempt: 1
I20260620 23:10:36.297252 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.296782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60866)
0620 23:10:36.296991 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.297095 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 23:10:36.298238 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.297841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60870)
0620 23:10:36.297980 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.298063 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 23:10:36.298434 2695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:45251: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
I20260620 23:10:36.366698 2695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } attempt: 2
I20260620 23:10:36.367981 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.367418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60886)
0620 23:10:36.367665 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.367781 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:10:36.369105 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.368651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60890)
0620 23:10:36.368810 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.368907 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:10:36.369324 2695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:45251: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
I20260620 23:10:36.460657 2695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } attempt: 3
I20260620 23:10:36.461906 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.461339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60892)
0620 23:10:36.461602 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.461724 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 23:10:36.463001 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.462537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60902)
0620 23:10:36.462689 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.462780 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:10:36.463228 2695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:45251: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
W20260620 23:10:36.576900 2702 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:10:36.577232 2702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:36.577306 2702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:36.577404 2702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:36.584606 2702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:36.584740 2702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:36.584852 2702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:36.584919 2702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:36.585002 2702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:36.594858 2702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:33375,127.12.2.125:45251,127.12.2.124:34017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.125.keytab
--principal=kudu/127.12.2.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.125:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.2.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:36.596210 2702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:36.597512 2702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:10:36.606453 2695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } attempt: 4
I20260620 23:10:36.607856 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.607165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60916)
0620 23:10:36.607536 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.607684 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246}
W20260620 23:10:36.608389 2710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:36.608707 2707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:36.608963 2708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:36.609084 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.608665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60918)
0620 23:10:36.608860 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.608947 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260620 23:10:36.609303 2695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.125:45251: Network error: Client connection negotiation failed: client connection to 127.12.2.125:45251: connect: Connection refused (error 111)
I20260620 23:10:36.609884 2702 server_base.cc:1034] running on GCE node
I20260620 23:10:36.610601 2702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:36.611948 2702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:36.613165 2702 hybrid_clock.cc:648] HybridClock initialized: now 1781997036613096 us; error 36 us; skew 500 ppm
I20260620 23:10:36.613468 2702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:36 dist-test-slave-7g8g krb5kdc[2615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997036, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:36.619325 2702 init.cc:377] Logged in from keytab as kudu/127.12.2.125@KRBTEST.COM (short username kudu)
I20260620 23:10:36.620702 2702 webserver.cc:460] Webserver started at http://127.12.2.125:42189/ using document root <none> and password file <none>
I20260620 23:10:36.621381 2702 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:36.621479 2702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:36.621773 2702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:36.624715 2702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/data/instance:
uuid: "5645ff73a58f458d82ecc161a8d88a69"
format_stamp: "Formatted at 2026-06-20 23:10:36 on dist-test-slave-7g8g"
server_key: "6bf296999f497e91fab5d8dc0486344e"
server_key_iv: "474c08bd534f3fa1e86c15af12e9c9ba"
server_key_version: "encryptionkey@0"
I20260620 23:10:36.625458 2702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/wal/instance:
uuid: "5645ff73a58f458d82ecc161a8d88a69"
format_stamp: "Formatted at 2026-06-20 23:10:36 on dist-test-slave-7g8g"
server_key: "6bf296999f497e91fab5d8dc0486344e"
server_key_iv: "474c08bd534f3fa1e86c15af12e9c9ba"
server_key_version: "encryptionkey@0"
I20260620 23:10:36.630374 2702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:10:36.633682 2717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:36.634904 2702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:10:36.635105 2702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/wal
uuid: "5645ff73a58f458d82ecc161a8d88a69"
format_stamp: "Formatted at 2026-06-20 23:10:36 on dist-test-slave-7g8g"
server_key: "6bf296999f497e91fab5d8dc0486344e"
server_key_iv: "474c08bd534f3fa1e86c15af12e9c9ba"
server_key_version: "encryptionkey@0"
I20260620 23:10:36.635295 2702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:36.649557 2702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:36.653235 2702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:36.653469 2702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:36.669934 2702 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.125:45251
I20260620 23:10:36.669958 2768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.125:45251 every 8 connection(s)
I20260620 23:10:36.671244 2702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/data/info.pb
I20260620 23:10:36.674656 2769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:36.679165 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 2702
I20260620 23:10:36.679327 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-1/wal/instance
I20260620 23:10:36.679745 12297 external_mini_cluster.cc:1428] Setting key 41d8bcb3b56354bbd09ff2f62eac1e64
I20260620 23:10:36.680929 2769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:10:36 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997036, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:10:36.701066 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.683408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.12.2.126:33375)
0620 23:10:36.683580 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:36.683585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:36.684765 (+ 1180us) server_negotiation.cc:407] Connection header received
0620 23:10:36.686544 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:36.686550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:36.686629 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:36.686716 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:36.688203 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.688859 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.690920 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.691435 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.691549 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:36.695369 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:36.695374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:36.695377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:36.695421 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:36.697886 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:36.698721 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:36.698725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:36.698727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:36.698843 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:36.699310 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:36.699314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:36.699316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:36.699657 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:36.699778 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:36.700662 (+ 884us) server_negotiation.cc:299] Negotiation successful
0620 23:10:36.700835 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:10:36.702502 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.683897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:33375 (local address 127.0.0.1:60676)
0620 23:10:36.684617 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.684653 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:36.686278 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:36.686973 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:36.686984 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:36.687479 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:36.688034 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:36.688058 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.689012 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.689018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:36.690729 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:36.690743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.691672 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.691678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:36.691917 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:36.692584 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:36.692614 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:36.695130 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:36.698020 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:36.698027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:36.698044 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:36.698608 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:36.698953 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:36.698958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:36.698961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:36.699181 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:36.699778 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:36.699785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:36.700220 (+ 435us) client_negotiation.cc:769] Sending connection context
0620 23:10:36.700523 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 23:10:36.701061 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":304,"threads_started":1}
I20260620 23:10:36.704358 2769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.2.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.2.124@KRBTEST.COM" created.
Jun 20 23:10:36 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997036, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
I20260620 23:10:36.717702 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.705139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60930)
0620 23:10:36.705325 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.705347 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:36.705489 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:36.706283 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:36.706286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:36.706629 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:36.706775 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:36.706784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.708145 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.708151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:36.708970 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:36.708981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.709747 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.709753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:36.709899 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:36.710412 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:36.710427 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:36.712283 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:36.715451 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:36.715455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:36.715460 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:36.715787 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:36.716115 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:36.716131 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:36.716134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:36.716244 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:36.716790 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:36.716795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:36.716958 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 23:10:36.717069 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 23:10:36.717329 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 23:10:36.717803 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.705286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60930 (local address 127.12.2.125:45251)
0620 23:10:36.705890 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:36.705897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:36.705921 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:10:36.705964 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:36.705969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:36.706139 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:36.706246 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:36.706955 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.708020 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.709087 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.709604 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.709705 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:36.712472 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:36.712482 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:36.712495 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:36.712522 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:36.715344 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:36.715899 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:36.715906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:36.715908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:36.716013 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:36.716334 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:36.716338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:36.716342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:36.716674 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:36.716852 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:36.717231 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 23:10:36.717436 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":354,"threads_started":1}
I20260620 23:10:36.720420 2769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:10:36.721556 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.721092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:32986)
0620 23:10:36.721277 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.721387 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:10:36.721834 2720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
I20260620 23:10:36.723415 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.723056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:32998)
0620 23:10:36.723197 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.723299 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:10:36.723649 2769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:34017: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.2.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab.
Entry for principal kudu/127.12.2.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab.
I20260620 23:10:36.735980 12297 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyvnGin/build/asan/bin/kudu
/tmp/dist-test-taskyvnGin/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.124:34017
--webserver_interface=127.12.2.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab
--principal=kudu/127.12.2.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.2.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.2.126:33375,127.12.2.125:45251,127.12.2.124:34017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:10:36.772975 2769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } attempt: 1
I20260620 23:10:36.774124 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.773656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33010)
0620 23:10:36.773845 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.773968 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:10:36.775228 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.774832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33024)
0620 23:10:36.775011 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.775112 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:10:36.775444 2769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:34017: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
I20260620 23:10:36.843643 2769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } attempt: 2
I20260620 23:10:36.844750 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.844317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33040)
0620 23:10:36.844507 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.844599 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 23:10:36.845674 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.845370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33042)
0620 23:10:36.845491 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.845564 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 23:10:36.845873 2769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:34017: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
I20260620 23:10:36.908614 2695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } attempt: 5
Jun 20 23:10:36 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997036, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
I20260620 23:10:36.922744 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.909320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60940)
0620 23:10:36.909506 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.909526 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:36.909646 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:36.910026 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:36.910030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:36.910430 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:36.910596 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:36.910621 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.912153 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.912157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:36.913016 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:36.913030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.913936 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.913940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:36.914076 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:36.914660 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:36.914677 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:36.916855 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:36.920549 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:36.920553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:36.920556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:36.920926 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:36.921334 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:36.921337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:36.921339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:36.921458 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:36.922043 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:36.922046 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:36.922185 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 23:10:36.922290 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:10:36.922488 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 23:10:36.923035 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.909432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.12.2.125:45251)
0620 23:10:36.909618 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:36.909624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:36.909648 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:10:36.909769 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:36.909774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:36.909877 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:36.909986 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:36.910826 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.912017 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.913165 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:36.913810 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:36.913935 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:36.917059 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:36.917065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:36.917070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:36.917111 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:36.920402 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:36.921054 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:36.921060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:36.921064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:36.921227 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:36.921551 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:36.921556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:36.921559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:36.921942 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:36.922112 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:36.922423 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 23:10:36.922668 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:10:36.923969 2695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:10:36.924973 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.924614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33054)
0620 23:10:36.924763 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.924837 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:10:36.925798 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.925492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33056)
0620 23:10:36.925635 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.925705 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 23:10:36.925982 2695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:34017: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
I20260620 23:10:36.937055 2769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } attempt: 3
I20260620 23:10:36.938141 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.937674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33062)
0620 23:10:36.937859 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.937959 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 23:10:36.939148 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.938685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33064)
0620 23:10:36.938895 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.939008 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260620 23:10:36.939342 2769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:34017: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
I20260620 23:10:36.977195 2695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } attempt: 1
I20260620 23:10:36.978221 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.977786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33066)
0620 23:10:36.977975 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.978065 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:10:36.979125 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:36.978788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33082)
0620 23:10:36.978925 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:36.979024 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 23:10:36.979306 2695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:34017: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
I20260620 23:10:37.047510 2695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } attempt: 2
I20260620 23:10:37.048655 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.048174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33094)
0620 23:10:37.048384 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:37.048476 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 23:10:37.049728 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.049343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33098)
0620 23:10:37.049548 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:37.049619 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 23:10:37.049932 2695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:34017: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
W20260620 23:10:37.053535 2776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:10:37.053874 2776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:37.053965 2776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:37.054078 2776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:37.061398 2776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:37.061504 2776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:37.061594 2776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:37.061657 2776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:37.061738 2776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:37.071719 2776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.2.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.2.126:33375,127.12.2.125:45251,127.12.2.124:34017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/krb5kdc/kudu_127.12.2.124.keytab
--principal=kudu/127.12.2.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.2.124:34017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.2.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:37.073031 2776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:37.074324 2776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:10:37.082564 2769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } attempt: 4
I20260620 23:10:37.083796 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.083271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33100)
0620 23:10:37.083502 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:37.083602 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 23:10:37.085111 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.084545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33112)
0620 23:10:37.084792 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:37.084894 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260620 23:10:37.085438 2769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.2.124:34017: Network error: Client connection negotiation failed: client connection to 127.12.2.124:34017: connect: Connection refused (error 111)
W20260620 23:10:37.085850 2781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:37.085892 2782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:37.086360 2784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:37.086875 2776 server_base.cc:1034] running on GCE node
I20260620 23:10:37.087733 2776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:37.089200 2776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:37.090402 2776 hybrid_clock.cc:648] HybridClock initialized: now 1781997037090335 us; error 41 us; skew 500 ppm
I20260620 23:10:37.090729 2776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:37 dist-test-slave-7g8g krb5kdc[2615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997037, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:37.093698 2776 init.cc:377] Logged in from keytab as kudu/127.12.2.124@KRBTEST.COM (short username kudu)
I20260620 23:10:37.095160 2776 webserver.cc:460] Webserver started at http://127.12.2.124:35165/ using document root <none> and password file <none>
I20260620 23:10:37.095844 2776 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:37.095932 2776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:37.096220 2776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:37.099246 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/data/instance:
uuid: "13a588eff1c84f4da9589add72aac77c"
format_stamp: "Formatted at 2026-06-20 23:10:37 on dist-test-slave-7g8g"
server_key: "56e05459ab8dae5b400fa936860e8b93"
server_key_iv: "4275c731b89ba04767da25077552173b"
server_key_version: "encryptionkey@0"
I20260620 23:10:37.099977 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/wal/instance:
uuid: "13a588eff1c84f4da9589add72aac77c"
format_stamp: "Formatted at 2026-06-20 23:10:37 on dist-test-slave-7g8g"
server_key: "56e05459ab8dae5b400fa936860e8b93"
server_key_iv: "4275c731b89ba04767da25077552173b"
server_key_version: "encryptionkey@0"
I20260620 23:10:37.105064 2776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:10:37.108417 2791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:37.109731 2776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:10:37.109869 2776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/data,/tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/wal
uuid: "13a588eff1c84f4da9589add72aac77c"
format_stamp: "Formatted at 2026-06-20 23:10:37 on dist-test-slave-7g8g"
server_key: "56e05459ab8dae5b400fa936860e8b93"
server_key_iv: "4275c731b89ba04767da25077552173b"
server_key_version: "encryptionkey@0"
I20260620 23:10:37.110046 2776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:37.124459 2776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:37.128093 2776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:37.128296 2776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:37.145182 2776 rpc_server.cc:307] RPC server started. Bound to: 127.12.2.124:34017
I20260620 23:10:37.145201 2842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.2.124:34017 every 8 connection(s)
I20260620 23:10:37.146486 2776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/data/info.pb
I20260620 23:10:37.149480 12297 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyvnGin/build/asan/bin/kudu as pid 2776
I20260620 23:10:37.149626 12297 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyvnGin/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996696601630-12297-0/minicluster-data/master-2/wal/instance
I20260620 23:10:37.150024 12297 external_mini_cluster.cc:1428] Setting key 7cca7e7381a784716a25831cac24a1b9
I20260620 23:10:37.150067 2843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:37.156137 2695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } attempt: 3
I20260620 23:10:37.159605 2843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:10:37 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
Jun 20 23:10:37 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997036, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.126@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
Jun 20 23:10:37 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997037, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:10:37.173352 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.156672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33126)
0620 23:10:37.156861 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:37.156876 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:37.156962 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:37.157714 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:37.157719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:37.158202 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:37.158380 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:37.158392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.159939 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.159945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:37.161287 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:37.161303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.162893 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.162899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:37.163087 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.163665 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:37.163681 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:37.166101 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:37.170084 (+ 3983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:37.170090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:37.170093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:37.170579 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:37.171050 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:37.171055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:37.171059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:37.171223 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:37.172209 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:37.172214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:37.172414 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 23:10:37.172594 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 23:10:37.172941 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"spinlock_wait_cycles":15744}
I20260620 23:10:37.174640 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.156801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.12.2.124:34017)
0620 23:10:37.157398 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.157403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.157424 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:10:37.157459 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.157463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.157521 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.157649 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.158572 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.159777 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.161453 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.162714 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.163376 (+ 662us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.166293 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:37.166299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:37.166302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:37.166332 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:37.169915 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.170742 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.170747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.170750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.170883 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.171327 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.171331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.171334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.172060 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:37.172250 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:37.172833 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 23:10:37.173240 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":474,"spinlock_wait_cycles":1920,"thread_start_us":324,"threads_started":1}
I20260620 23:10:37.175848 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.151617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.12.2.124:34017)
0620 23:10:37.152325 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.152340 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.152378 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:10:37.154317 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.154338 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.154723 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.155084 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.157444 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.159133 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.160540 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.161177 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.161343 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.165931 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:37.165938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:37.165949 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:37.165985 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:37.170030 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.170689 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.170698 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.170706 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.170874 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.171275 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.171281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.171284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.172061 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:37.172346 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:37.175390 (+ 3044us) server_negotiation.cc:299] Negotiation successful
0620 23:10:37.175554 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":187,"server-negotiator.queue_time_us":521,"spinlock_wait_cycles":57216,"thread_start_us":333,"threads_started":1}
I20260620 23:10:37.178256 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.162374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.12.2.126:33375)
0620 23:10:37.162580 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.162587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.163237 (+ 650us) server_negotiation.cc:407] Connection header received
0620 23:10:37.163777 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.163782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.163850 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.163952 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.165022 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.165894 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.167942 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.168602 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.168726 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.172394 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:37.172401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:37.172406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:37.172441 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:37.175200 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.176013 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.176018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.176022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.176171 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.176611 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.176616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.176618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.177004 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:37.177145 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:37.177691 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 23:10:37.177909 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:10:37.179203 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.162301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.126:33375 (local address 127.0.0.1:60680)
0620 23:10:37.163111 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:37.163154 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:37.163651 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:37.164020 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:37.164027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:37.164630 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:37.164848 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:37.164859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.166072 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.166077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:37.167804 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:37.167822 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.168802 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.168809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:37.168984 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.169653 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:37.169682 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:37.172161 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:37.175310 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:37.175318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:37.175324 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:37.175883 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:37.176284 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:37.176287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:37.176290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:37.176471 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:37.177210 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:37.177217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:37.177382 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 23:10:37.178499 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0620 23:10:37.178804 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":429,"threads_started":1}
I20260620 23:10:37.180487 2843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:10:37.182421 2695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159: Bootstrap starting.
I20260620 23:10:37.185864 2695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:37.187249 2695 log.cc:824] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:10:37 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997037, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
Jun 20 23:10:37 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.126@KRBTEST.COM
I20260620 23:10:37.195013 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.181202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.125:45251 (local address 127.0.0.1:60944)
0620 23:10:37.181368 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:37.181390 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:37.181498 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:37.181775 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:37.181780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:37.182133 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:37.182299 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:37.182311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.183746 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.183751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:37.184740 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:37.184751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.185577 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.185583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:37.185763 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.186285 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:37.186307 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:37.188559 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:37.192600 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:37.192605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:37.192610 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:37.192974 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:37.193354 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:37.193359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:37.193362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:37.193519 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:37.194072 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:37.194078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:37.194230 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 23:10:37.194380 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 23:10:37.194632 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:10:37.195072 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.181169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.12.2.125:45251)
0620 23:10:37.181342 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.181347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.181462 (+ 115us) server_negotiation.cc:407] Connection header received
0620 23:10:37.181596 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.181602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.181670 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.181744 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.182473 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.183554 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.184868 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.185278 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.185384 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.188774 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:37.188781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:37.188786 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:37.188828 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:37.192467 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.193067 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.193072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.193076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.193230 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.193615 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.193620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.193624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.193934 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:37.194090 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:37.194504 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 23:10:37.194723 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:10:37.196208 2843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:10:37.196467 2695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159: No bootstrap required, opened a new log
I20260620 23:10:37.199157 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.182346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.12.2.126:33375)
0620 23:10:37.182493 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.182498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.182515 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:10:37.182571 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.182575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.182637 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.182738 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.184148 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.185234 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.186903 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.187539 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.187672 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.191988 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:37.191993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:37.191996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:37.192025 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:37.195768 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.196524 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.196529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.196532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.196693 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.197195 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.197210 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.197214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.197571 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:37.197829 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:37.198698 (+ 869us) server_negotiation.cc:299] Negotiation successful
0620 23:10:37.198859 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 23:10:37 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997037, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.124@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
I20260620 23:10:37.205307 2695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } }
I20260620 23:10:37.205583 2695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d97ee62bf8451fbf2d91150f01f159, State: Initialized, Role: FOLLOWER
I20260620 23:10:37.205957 2695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } }
I20260620 23:10:37.207190 2850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } } }
I20260620 23:10:37.207707 2850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:37.208294 2695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:37.211170 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.196867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33136)
0620 23:10:37.197134 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:37.197152 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:37.197234 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:37.197502 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:37.197507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:37.197937 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:37.198116 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:37.198126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.199979 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.199984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:37.201094 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:37.201107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.202067 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.202073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:37.202239 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.202794 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:37.202818 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:37.205272 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:37.208647 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:37.208652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:37.208656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:37.209114 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:37.209494 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:37.209498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:37.209502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:37.209633 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:37.210269 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:37.210273 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:37.210412 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 23:10:37.210527 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 23:10:37.210790 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
Jun 20 23:10:37 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.2.125@KRBTEST.COM
I20260620 23:10:37.214227 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.197083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.12.2.124:34017)
0620 23:10:37.197237 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.197242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.197259 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:10:37.197309 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.197314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.197377 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.197466 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.198251 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.199850 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.201196 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.201778 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.201902 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.205453 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:37.205458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:37.205463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:37.205497 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:37.208525 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.209232 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.209237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.209241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.209373 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.209724 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.209728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.209731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.210104 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:37.210229 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:37.213682 (+ 3453us) server_negotiation.cc:299] Negotiation successful
0620 23:10:37.213927 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:10:37.220046 2843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c: Bootstrap starting.
I20260620 23:10:37.222184 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.200921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.12.2.125:45251)
0620 23:10:37.201073 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.201078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.201096 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:10:37.201142 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.201147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.201212 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.201310 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.202763 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.203876 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.208955 (+ 5079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.209566 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.209697 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.215392 (+ 5695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:37.215399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:37.215403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:37.215442 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:37.218613 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.219424 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.219429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.219432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.219586 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.220020 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.220025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.220028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.220345 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:37.220610 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:37.221640 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0620 23:10:37.221866 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:10:37.224004 2843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:37.225241 2843 log.cc:824] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:37.227319 12297 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:10:37.230569 2670 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "06d97ee62bf8451fbf2d91150f01f159"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60684
I20260620 23:10:37.231030 2670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:10:37.231242 2670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:37.231603 2843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c: No bootstrap required, opened a new log
I20260620 23:10:37.234920 2670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } }
I20260620 23:10:37.238314 2670 leader_election.cc:288] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [CANDIDATE]: Term 1 election: Requested vote from peers 5645ff73a58f458d82ecc161a8d88a69 (127.12.2.125:45251), 13a588eff1c84f4da9589add72aac77c (127.12.2.124:34017)
I20260620 23:10:37.228699 2628 master_runner.cc:186] Error getting master registration for 127.12.2.125:45251: OK, instance_id { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" instance_seqno: 1781997036665190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:10:37.239778 2628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 23:10:37.241281 2861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:10:37.241406 2861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:10:37.245582 2843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } }
I20260620 23:10:37.246007 2843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a588eff1c84f4da9589add72aac77c, State: Initialized, Role: FOLLOWER
I20260620 23:10:37.246508 2843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } }
I20260620 23:10:37.267580 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } } }
I20260620 23:10:37.268060 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:37.269006 2843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:37.269189 2818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06d97ee62bf8451fbf2d91150f01f159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "13a588eff1c84f4da9589add72aac77c"
I20260620 23:10:37.269788 2818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:37.273554 2818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06d97ee62bf8451fbf2d91150f01f159 in term 1.
I20260620 23:10:37.274482 2646 leader_election.cc:302] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d97ee62bf8451fbf2d91150f01f159, 13a588eff1c84f4da9589add72aac77c; no voters:
I20260620 23:10:37.275030 2850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:37.298700 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.257151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.12.2.125:45251)
0620 23:10:37.257326 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.257332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.268038 (+ 10706us) server_negotiation.cc:407] Connection header received
0620 23:10:37.271415 (+ 3377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.271422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.271509 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.271580 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.275033 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.276147 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.281641 (+ 5494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.282373 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.282469 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.288176 (+ 5707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:37.288183 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:37.288189 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:37.288250 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:37.291748 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.296146 (+ 4398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.296152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.296156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.296344 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.296827 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.296832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.296835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.297246 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:37.297493 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:37.298077 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 23:10:37.298330 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:10:37.316907 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.256736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.12.2.126:33375)
0620 23:10:37.256946 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.256954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.268514 (+ 11560us) server_negotiation.cc:407] Connection header received
0620 23:10:37.269099 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.269106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.269193 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.269267 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.273642 (+ 4375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.274801 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.279895 (+ 5094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.280499 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.280606 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.292669 (+ 12063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:37.292676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:37.292681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:37.292726 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:37.313108 (+ 20382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.313965 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.313971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.313975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.314157 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.314598 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.314603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.314606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.314965 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:37.315193 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:37.315742 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 23:10:37.315980 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:10:37.317576 2850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [term 1 LEADER]: Becoming Leader. State: Replica: 06d97ee62bf8451fbf2d91150f01f159, State: Running, Role: LEADER
I20260620 23:10:37.318101 2850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } }
I20260620 23:10:37.329317 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d97ee62bf8451fbf2d91150f01f159. Latest consensus state: current_term: 1 leader_uuid: "06d97ee62bf8451fbf2d91150f01f159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } } }
I20260620 23:10:37.329699 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:37.330586 2880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:37.332767 2880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:37.334761 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.268698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.12.2.124:34017)
0620 23:10:37.268855 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.268861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.308938 (+ 40077us) server_negotiation.cc:407] Connection header received
0620 23:10:37.309220 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.309227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.309319 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.309390 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.311057 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.312281 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.319666 (+ 7385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.320337 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.320439 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.324124 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:37.324131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:37.324136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:37.324180 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:37.327762 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.329431 (+ 1669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.329436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.329439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.329671 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.330132 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.330138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.330142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.330912 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:37.331733 (+ 821us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:37.332390 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 23:10:37.333314 (+ 924us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:10:37.352197 2818 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [term 1 FOLLOWER]: Refusing update from remote peer 06d97ee62bf8451fbf2d91150f01f159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:37.353250 2872 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:10:37.369167 2886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:10:37.369323 2886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:10:37.353200 2776 master_runner.cc:186] Error getting master registration for 127.12.2.125:45251: OK, instance_id { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" instance_seqno: 1781997036665190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:10:37.370733 2776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:10:37.375978 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d97ee62bf8451fbf2d91150f01f159. Latest consensus state: current_term: 1 leader_uuid: "06d97ee62bf8451fbf2d91150f01f159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } } }
I20260620 23:10:37.376287 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:37.384644 2769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } attempt: 5
I20260620 23:10:37.391788 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06d97ee62bf8451fbf2d91150f01f159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } } }
I20260620 23:10:37.392076 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 23:10:37 dist-test-slave-7g8g krb5kdc[2615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997036, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.2.125@KRBTEST.COM for kudu/127.12.2.124@KRBTEST.COM
I20260620 23:10:37.395659 2880 catalog_manager.cc:1347] Generated new cluster ID: 603a3e7cb61040e1968f06187404bf18
I20260620 23:10:37.395763 2880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:37.409102 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06d97ee62bf8451fbf2d91150f01f159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } } }
I20260620 23:10:37.409405 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:37.410339 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.385221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.2.124:34017 (local address 127.0.0.1:33156)
0620 23:10:37.385556 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:37.385576 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:37.385690 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:37.387509 (+ 1819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:37.387514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:37.388019 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:37.388212 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:37.388224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.389643 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.389649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:37.391089 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:37.391105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.392277 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.392283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:37.392480 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.393131 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:37.393156 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:37.403422 (+ 10266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:37.407525 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:37.407531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:37.407535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:37.408057 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:37.408511 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:37.408516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:37.408520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:37.408679 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:37.409368 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:37.409373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:37.409528 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:10:37.409668 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:10:37.409973 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260620 23:10:37.411289 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:37.385900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.12.2.124:34017)
0620 23:10:37.386067 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.386072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.386089 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:10:37.386140 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.386145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.386242 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.386334 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.388375 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.389477 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.391251 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:37.391957 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:37.392076 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:37.403665 (+ 11589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:37.403671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:37.403677 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:37.403715 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:37.407351 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.408201 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.408207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.408210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.408375 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:37.408805 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:37.408811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:37.408814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:37.409170 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:37.409306 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:37.410692 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0620 23:10:37.410925 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:10:37.417090 2769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69: Bootstrap starting.
I20260620 23:10:37.422574 2769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:37.423934 2769 log.cc:824] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:37.422900 2880 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:37.428596 2880 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:37.443195 2769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69: No bootstrap required, opened a new log
I20260620 23:10:37.455413 2769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } }
I20260620 23:10:37.455873 2769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5645ff73a58f458d82ecc161a8d88a69, State: Initialized, Role: FOLLOWER
I20260620 23:10:37.456537 2769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } }
I20260620 23:10:37.467013 2769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:37.467402 2743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:37.464630 2893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } } }
I20260620 23:10:37.468648 2893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:37.471225 2744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "06d97ee62bf8451fbf2d91150f01f159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5645ff73a58f458d82ecc161a8d88a69"
I20260620 23:10:37.473489 2743 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [term 1 FOLLOWER]: Refusing update from remote peer 06d97ee62bf8451fbf2d91150f01f159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:37.476967 2744 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06d97ee62bf8451fbf2d91150f01f159 in term 1.
I20260620 23:10:37.481719 2850 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:37.482787 2880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159: Generated new TSK 0
I20260620 23:10:37.483914 2880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:10:37.520308 2893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d97ee62bf8451fbf2d91150f01f159. Latest consensus state: current_term: 1 leader_uuid: "06d97ee62bf8451fbf2d91150f01f159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } } }
I20260620 23:10:37.520627 2893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:37.521006 2902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06d97ee62bf8451fbf2d91150f01f159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } } }
I20260620 23:10:37.521330 2902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:10:37.550984 2911 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 23:10:37.587152 2698 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.573131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.12.2.126:33375)
0620 23:10:37.573313 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.573319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.585223 (+ 11904us) server_negotiation.cc:407] Connection header received
0620 23:10:37.585534 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.585541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.585627 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.585699 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.586862 (+ 1163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60694: BlockingRecv error: recv got EOF from 127.0.0.1:60694 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 23:10:37.588244 2773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.573232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.12.2.125:45251)
0620 23:10:37.573391 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.573396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.585325 (+ 11929us) server_negotiation.cc:407] Connection header received
0620 23:10:37.586711 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.586717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.586808 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.586873 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.588012 (+ 1139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60984: BlockingRecv error: recv got EOF from 127.0.0.1:60984 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 23:10:37.589521 2846 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.585864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.12.2.124:34017)
0620 23:10:37.586033 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.586049 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.587894 (+ 1845us) server_negotiation.cc:407] Connection header received
0620 23:10:37.588100 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.588105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.588180 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.588245 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.589242 (+ 997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33172: BlockingRecv error: recv got EOF from 127.0.0.1:33172 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 23:10:37.609347 2773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.600481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.12.2.125:45251)
0620 23:10:37.600712 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.600718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.607517 (+ 6799us) server_negotiation.cc:407] Connection header received
0620 23:10:37.607739 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.607745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.607876 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.608129 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.609099 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60990: BlockingRecv error: recv got EOF from 127.0.0.1:60990 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 23:10:37.652808 2773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.640109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.12.2.125:45251)
0620 23:10:37.640289 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.640296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.651254 (+ 10958us) server_negotiation.cc:407] Connection header received
0620 23:10:37.651482 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.651488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.651590 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.651690 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.652593 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60994: BlockingRecv error: recv got EOF from 127.0.0.1:60994 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 23:10:37.661068 2773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.658999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.12.2.125:45251)
0620 23:10:37.659180 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.659186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.659665 (+ 479us) server_negotiation.cc:407] Connection header received
0620 23:10:37.659868 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.659874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.659978 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.660084 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.660837 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60996: BlockingRecv error: recv got EOF from 127.0.0.1:60996 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 23:10:37.671303 2773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.668831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.12.2.125:45251)
0620 23:10:37.669001 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.669008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.669543 (+ 535us) server_negotiation.cc:407] Connection header received
0620 23:10:37.669764 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.669770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.669872 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.669980 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.671073 (+ 1093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32774: BlockingRecv error: recv got EOF from 127.0.0.1:32774 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 23:10:37.701974 2773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.699601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.12.2.125:45251)
0620 23:10:37.699803 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.699809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.700336 (+ 527us) server_negotiation.cc:407] Connection header received
0620 23:10:37.700644 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.700650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.700750 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.700865 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.701737 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32778: BlockingRecv error: recv got EOF from 127.0.0.1:32778 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 23:10:37.724941 2773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.722361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.12.2.125:45251)
0620 23:10:37.722567 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.722573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.723156 (+ 583us) server_negotiation.cc:407] Connection header received
0620 23:10:37.723397 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.723417 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.723521 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.723602 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.724708 (+ 1106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32788: BlockingRecv error: recv got EOF from 127.0.0.1:32788 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 23:10:37.767225 2773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.764779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.12.2.125:45251)
0620 23:10:37.765018 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.765025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.765585 (+ 560us) server_negotiation.cc:407] Connection header received
0620 23:10:37.765765 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.765772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.765877 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.765995 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.766902 (+ 907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32794: BlockingRecv error: recv got EOF from 127.0.0.1:32794 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 23:10:37.836531 2773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.834737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.12.2.125:45251)
0620 23:10:37.835001 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.835008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.835240 (+ 232us) server_negotiation.cc:407] Connection header received
0620 23:10:37.835437 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.835442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.835521 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.835611 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.836359 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49882: BlockingRecv error: recv got EOF from 127.0.0.1:49882 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 23:10:37.970003 2773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:37.968056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.12.2.125:45251)
0620 23:10:37.968250 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:37.968256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:37.968635 (+ 379us) server_negotiation.cc:407] Connection header received
0620 23:10:37.968848 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:37.968852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:37.968925 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:37.969028 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:37.969778 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49898: BlockingRecv error: recv got EOF from 127.0.0.1:49898 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:10:38.011545 2850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06d97ee62bf8451fbf2d91150f01f159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d97ee62bf8451fbf2d91150f01f159" member_type: VOTER last_known_addr { host: "127.12.2.126" port: 33375 } } peers { permanent_uuid: "5645ff73a58f458d82ecc161a8d88a69" member_type: VOTER last_known_addr { host: "127.12.2.125" port: 45251 } } peers { permanent_uuid: "13a588eff1c84f4da9589add72aac77c" member_type: VOTER last_known_addr { host: "127.12.2.124" port: 34017 } } }
I20260620 23:10:38.011839 2850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d97ee62bf8451fbf2d91150f01f159 [sys.catalog]: This master's current role is: LEADER
W20260620 23:10:38.231217 2773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:10:38.229464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.12.2.125:45251)
0620 23:10:38.229674 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:38.229680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:38.229860 (+ 180us) server_negotiation.cc:407] Connection header received
0620 23:10:38.230115 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:38.230120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:38.230190 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:38.230273 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:38.231059 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49910: BlockingRecv error: recv got EOF from 127.0.0.1:49910 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:10:38.371516 2886 catalog_manager.cc:1259] Loaded cluster ID: 603a3e7cb61040e1968f06187404bf18
I20260620 23:10:38.371630 2886 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c: loading cluster ID for follower catalog manager: success
I20260620 23:10:38.374179 2886 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c: acquiring CA information for follower catalog manager: success
I20260620 23:10:38.376087 2886 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13a588eff1c84f4da9589add72aac77c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:10:38.555296 2911 catalog_manager.cc:1259] Loaded cluster ID: 603a3e7cb61040e1968f06187404bf18
I20260620 23:10:38.555418 2911 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69: loading cluster ID for follower catalog manager: success
I20260620 23:10:38.557912 2911 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69: acquiring CA information for follower catalog manager: success
I20260620 23:10:38.559667 2911 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5645ff73a58f458d82ecc161a8d88a69: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:10:38.752637 2978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:38.746769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49926 (local address 127.12.2.125:45251)
0620 23:10:38.747510 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:38.747514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:38.747529 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:10:38.747593 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:38.747597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:38.747724 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:38.747828 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:38.748281 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:38.749117 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:38.750425 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:38.750864 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:38.751021 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:38.751483 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:38.752020 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:38.752151 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:38.752257 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 23:10:38.752335 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":469,"threads_started":1}
I20260620 23:10:38.765296 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:38.758997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.12.2.124:34017)
0620 23:10:38.759507 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:38.759513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:38.759810 (+ 297us) server_negotiation.cc:407] Connection header received
0620 23:10:38.760021 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:38.760025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:38.760166 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:38.760292 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:38.760704 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:38.761364 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:38.763202 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:38.763582 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:38.763673 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:38.764209 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:38.764717 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:38.764817 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:38.764943 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 23:10:38.765006 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":227,"threads_started":1}
I20260620 23:10:38.783087 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:38.773042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.12.2.126:33375)
0620 23:10:38.775821 (+ 2779us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:38.775827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:38.775847 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:10:38.775954 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:38.775960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:38.776188 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:38.776368 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:38.776953 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:38.778073 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:38.780287 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:38.780929 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:38.781560 (+ 631us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:38.781739 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:38.782123 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:38.782611 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:38.782665 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 23:10:38.782744 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2656,"thread_start_us":2559,"threads_started":1}
I20260620 23:10:38.783206 2978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:38.772775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.12.2.125:45251)
0620 23:10:38.772938 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:38.772944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:38.775472 (+ 2528us) server_negotiation.cc:407] Connection header received
0620 23:10:38.776796 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:38.776801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:38.776890 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:38.776958 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:38.777775 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:38.778879 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:38.781048 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:38.781685 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:38.781776 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:38.782370 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:38.782490 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:38.782569 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:38.782806 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:10:38.782886 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:10:38.787326 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 2628
I20260620 23:10:38.788058 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:38.773876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.12.2.124:34017)
0620 23:10:38.774034 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:38.774040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:38.781591 (+ 7551us) server_negotiation.cc:407] Connection header received
0620 23:10:38.781977 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:38.781982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:38.782069 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:38.782123 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:38.782909 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:38.783907 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:38.786336 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:38.786716 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:38.786817 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:38.787365 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:38.787484 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:38.787582 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:38.787708 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 23:10:38.787786 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 23:10:38.788229 2690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:38.896854 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 2628
I20260620 23:10:38.913997 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 2702
I20260620 23:10:38.915129 2764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:39.018432 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 2702
I20260620 23:10:39.034180 12297 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskyvnGin/build/asan/bin/kudu pid 2776
I20260620 23:10:39.035207 2838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:39.138118 12297 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyvnGin/build/asan/bin/kudu with pid 2776
2026-06-20T23:10:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4607 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9287 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (342500 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST